Series · roadmap stage 3/13
The Data Processing Series
Ten lessons in order, from the mental model to a capstone checklist. Every lesson is structured by us and grounded in a real, cited source from the registry — no invented commands.
10 of 10 lessons published
- Lesson 1 The batch processing mental model
Batch processing in partition terms. Learn what a partition, a shuffle and a spill are, why "the job is slow" resolves into four named causes — skew, partition count, spill, over-scan — and why the write side carries one invariant: same dat
- Lesson 2 Start a PySpark project from the template
Scaffold a testable PySpark project with one command and learn where each piece lives: transforms in src/, 3-10 row fixtures in tests/data/, and a Makefile that runs lint plus tests offline. Finish by hardening the first transform with row-
- Lesson 3 Diagnose a skewed Spark stage
One Spark stage hangs while other executors idle. Open the stage view, read the p50-vs-p99 task durations, and follow the skew skill's decision order: AQE skew join first, salting second, broadcast only under a size budget — and never repar
- Lesson 4 Aggregations and windows without the skew
Aggregations and windows are shuffles in disguise. Treat the group-by key as a skew suspect, read p50 vs p99 and the spill bytes, then apply the matching fix — a salted two-stage aggregate, more partitions, or a partition predicate — and ne
- Lesson 5 Incremental processing with a watermark
Full reloads stop scaling once a table grows past millions of rows. Process a bounded window instead: a watermark ends it, a lookback absorbs late files, and an idempotent write (partition overwrite or MERGE) makes every re-run safe — inclu
- Lesson 6 Tune a slow job — memory, partitions, baselines
Run the tuning loop on a slow batch job: parse task-duration and shuffle metrics, classify the stage as skew, partition count, spill or scan, apply the single highest-impact change, and verify it against a stored baseline the CI guard enfor
- Lesson 7 Read a Spark event log with the Spark Lens plugin
Install the Spark Lens plugin and work its three commands over a Spark event log: /spark-analyze to build a task histogram and flag skew, /spark-explain-plan to check the join strategy, and /spark-baseline to store the cost baseline the gua
- Lesson 8 Review a PySpark transform for silent quality failures
Run the five-check pass over a PySpark transform before it merges: null propagation, join fan-out, silent coercion, non-determinism, and missing post-write assertions. Each finding lands as severity, file:line, rule and a suggested patch.
- Lesson 9 The 3am batch incident path
The nightly job is crawling or dead under memory pressure or skew. Follow the incident path: collect the four pieces of evidence, read p50 vs p99 and the spill bytes, decide revert vs fix, escalate on golden-hour SLAs, and record the skew/s
- Lesson 10 Capstone — the batch pipeline checklist
Bring the API → S3 → Spark → Parquet starter up end to end, then audit it: offline transform tests, idempotent partition-scoped writes, complete partitions, replay-safe extraction, and baseline metrics with a CI guard. One pipeline that exe