Batch Processing

The batch processing mental model

• Name the four causes of a slow batch job: skew, partition count, spill, over-scan
• Explain what a shuffle does to a hot key

Capstone — the batch pipeline checklist

Bring the API → S3 → Spark → Parquet starter up end to end, then audit it: offline transform tests, idempotent partition-scoped writes, complete partitions, replay-safe extraction, and baseline metrics with a CI guard.

Incremental processing with a watermark

• Bound incremental work with a watermark and a lookback window
• Write incremental output idempotently so a re-run is safe

Missing partition

An expected date partition is absent, so a downstream report silently excludes a day.

Idempotent Partition Overwrite

Retries and backfills duplicate or corrupt data because writes are appends.

Incremental Batch with a Watermark

Full reloads are too slow and too expensive once a table grows past millions of rows.