DE//AGENT
Explore
Tutorial Lesson 10 of 10 · The Data Processing Series community intermediate

Capstone — the batch pipeline checklist

Bring the API → S3 → Spark → Parquet starter up end to end, then audit it: offline transform tests, idempotent partition-scoped writes, complete partitions, replay-safe extraction, and baseline metrics with a CI guard. One pipeline that exercises every lesson in the series.

At a glance

When You finished the series and want one pipeline that exercises every lesson; A new batch pipeline needs a production checklist before its first real run
Where Batch Processing
Why A pipeline is done when re-runs are safe, partitions are complete, and the metrics that would catch today's failure are baselined.

Learning objectives

  • Run a batch pipeline end to end with offline tests
  • Prove idempotency with a two-run row-count check
  • Ship the completeness, hardening, and baseline guards

What you need

  • Docker (the starter runs MinIO + a local Spark)
  • An API endpoint to extract from, or the seeded sample data

Steps

  1. The pipeline — REST API → extract (paginate + retry) → S3 raw/ (JSON, immutable) → PySpark transform (type, dedupe, partition) → S3 curated/ (Parquet, dt=.., Hive layout). (from project-api-s3-spark-parquet )
  2. Bring it up — `make up` starts MinIO + a local Spark, `make seed` uploads sample raw JSON, `make test` runs the transform tests offline. (from project-api-s3-spark-parquet )
  3. The tests that matter — row count stable across two runs (idempotency), no nulls in the business key, and partition dt matches the logical date, not the wall clock. (from project-api-s3-spark-parquet )
  4. A re-run for the same date must not duplicate rows — the overwrite is partition-scoped. (from project-api-s3-spark-parquet )
  5. The invariant — running the same date twice yields identical output. (from pattern-idempotent-partition-overwrite )
  6. Harden — retry with jittered exponential backoff, alert on extract lag and on null business keys, checksum raw files, and store the API's cursor for replay. (from project-api-s3-spark-parquet )
  7. A completeness check asserts every expected partition exists, and the logical date is stored explicitly as a partition column, not derived at read time. (from runbook-missing-partition )

Check yourself

  • Two consecutive runs produce the same row count, and dt equals the logical date.
  • Baseline shuffle/spill/scan metrics exist and CI fails on a regression.
  • The overwrite is partition-scoped — a re-run for the same date does not duplicate rows.

Worked example

Bring the pipeline up — from project-api-s3-spark-parquet 

make up    # starts MinIO + a local Spark
make seed  # uploads sample raw JSON
make test  # runs the transform tests offline

The pipeline under test — from project-api-s3-spark-parquet 

REST API ─▶ [extract: paginate + retry] ─▶ S3 raw/ (JSON, immutable)
                                                  │
                                                  ▼
                                [PySpark transform: type, dedupe, partition]
                                                  │
                                                  ▼
                                S3 curated/ (Parquet, dt=.., Hive layout)

Failure scenarios to drill — from project-api-s3-spark-parquet 

- API returns 429 → backoff then resume from the last cursor, not from page 1
- A page arrives with a changed field type → quarantine the page, alert
- A re-run for the same date duplicates rows → the overwrite is not partition-scoped

Sourced from

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

  • A re-run for the same date duplicates rows → the overwrite is not partition-scoped.
  • An API returns 429 → backoff then resume from the last cursor, not from page 1.
  • An empty partition and an absent partition both read as 'no data' downstream.

Production considerations

  • Checksum raw files; store the API's cursor for replay.
  • Store the logical date explicitly as a partition column, not derived at read time.

Diagram

capstone checklist
   |
   |- run --> make up / seed / test (offline, MinIO + local Spark)
   |
   |- idempotency --> row count stable across two runs
   |
   |- completeness --> every expected dt partition exists
   |
   |- hardening --> backoff, cursor replay, raw checksums
   |
   |- guard --> baseline shuffle/spill/scan, CI fails on regression

Registry stats

Verification community · Oct 21, 2026
Completeness 81
Trust multiplier ×1.25
Composite 34.15 · adequate
Freshness 3 days since last source check
Difficulty intermediate
Version 0.1.0
License CC-BY-4.0
Security flags none