Apache Parquet

DE//AGENT currently lists 4 resources under Apache Parquet: 3 projects, 1 skill. Everything below is derived from the live registry — counts, capabilities, verification state and security posture come from the curated records, never from marketing copy.

Documented learning objectives across these resources:

• Express a quality contract as executable checks — via Pipeline → DQ → quarantine → publish
• Design a quarantine table with a reason — via Pipeline → DQ → quarantine → publish
• Alert on the quarantine rate, not just on failure — via Pipeline → DQ → quarantine → publish
• Test a transform without a Spark cluster — via Pipeline Test Author
• Assert invariants instead of golden files — via Pipeline Test Author
• Make a pipeline test deterministic — via Pipeline Test Author
• Handle pagination, retries and rate limits correctly — via API → S3 → Spark → Parquet
• Write an idempotent partition overwrite — via API → S3 → Spark → Parquet

4 more objectives live on the individual resource pages.

These resources map to stages: Reliability (1), Data Processing (1), Data Quality (1), Cloud (1). Follow the four-phase roadmap to place this topic inside a progression instead of learning it in isolation.

Every entry, its shape and its trust state — sorted by the registry's composite score. This table is the honest version of a marketing comparison: no sponsored placements, no unverifiable claims.

How scores are computed: source quality score.

Highest composite score first:

1. Pipeline → DQ → quarantine → publish score 38.65 — intermediate
2. Pipeline Test Author score 38 — intermediate
3. API → S3 → Spark → Parquet score 32.15 — intermediate

Scores rank provenance, completeness and trust — read how verification works before adopting anything.

Security-flagged entries: Pipeline → DQ → quarantine → publish (2 flags), Pipeline Test Author (2 flags), API → S3 → Spark → Parquet (5 flags), S3 → Glue → Athena (5 flags). Flags mean the record declares capabilities like credential use or write access — read the security posture on each page before connecting.

• Pipeline → DQ → quarantine → publish — Put a contract in front of a table: validate each batch, publish the clean rows, quarantine the rest with reasons, and alert on the quarantine rate. details
• Pipeline Test Author — Builds a local DuckDB/pyarrow harness with tiny fixtures so pipeline logic is testable in CI without a cluster, and asserts on invariants, not snapshots. details
• API → S3 → Spark → Parquet — Land a paginated REST API into S3 raw, transform with PySpark into partitioned Parquet, and make the whole thing idempotent and testable offline. details
• S3 → Glue → Athena — A serverless, query-in-place pattern: land Parquet in S3, catalogue it with Glue, and query with Athena while keeping scanned bytes (and cost) low. details

Real install commands from the registry — read the source before running any of them:

npx de-agent-platform add pipeline-test-author

Pipeline Test Author — review first

Official source: Apache Parquet (license: Apache-2.0) — start there for the canonical documentation, then come back for the vetted registry entries.

Adjacent topics: Python (3 shared), DuckDB (2 shared), Apache Spark (2 shared), Amazon S3 (2 shared).

Study path: the roadmap places this topic in a progression · the scoring model explains the numbers · the trust rule catalog shows every security check these entries passed through.