Roadmap · stage 3/13
Data Processing
Stage 3 of 13 · 26 resources · 5 categories
What this stage is
Data Processing is stage 3 of 13 in the DE//AGENT roadmap. It currently holds 26 resources: 10 tutorials, 4 references, 3 runbooks across 5 categories.
What it teaches
Batch engines, file formats, partitioning and incremental loads.
What you can do after this stage
- Choose a file format and partition layout
- Write an idempotent incremental batch job
Exit gate
Gate: You can build and re-run a batch pipeline safely. The stage deliverable: project-api-s3-spark-parquet.
Gates are self-assessed — the roadmap does not track your progress; it tells you what to be able to do before moving on.
Categories in this stage
Databases (4), Spark / PySpark (14), Batch Processing (6), APIs & Ingestion (1), File & Partner Ingestion (1)
Registry resources for this stage
Sorted by composite score — provenance, completeness and trust, per the scoring model:
| Resource | Type | Difficulty | Verification | Score |
|---|---|---|---|---|
| PostgreSQL MCP Server | MCP server | beginner | community | 44.5 |
| Spark Data Quality Review | Skill | intermediate | community | 41 |
| Spark Skew Diagnosis | Skill | advanced | community | 38.5 |
| Incremental Batch with a Watermark | Pattern | intermediate | community | 37.25 |
| Spark job is slow | Runbook | advanced | community | 36.75 |
| Idempotent Partition Overwrite | Pattern | beginner | community | 34.75 |
| Spark Optimizer | Agent | advanced | community | 34.5 |
| Start a PySpark project from the template | Tutorial | beginner | community | 34.15 |
| The batch processing mental model | Tutorial | beginner | community | 34.15 |
| Aggregations and windows without the skew | Tutorial | intermediate | community | 34.15 |
| Capstone — the batch pipeline checklist | Tutorial | intermediate | community | 34.15 |
| Diagnose a skewed Spark stage | Tutorial | intermediate | community | 34.15 |
| Incremental processing with a watermark | Tutorial | intermediate | community | 34.15 |
| Read a Spark event log with the Spark Lens plugin | Tutorial | intermediate | community | 34.15 |
| Review a PySpark transform for silent quality failures | Tutorial | intermediate | community | 34.15 |
| The 3am batch incident path | Tutorial | advanced | community | 34.15 |
| Tune a slow job — memory, partitions, baselines | Tutorial | advanced | community | 34.15 |
| Spark Lens | Plugin | advanced | community | 34.1 |
| PySpark Project Template | Template | intermediate | community | 34.1 |
| API → S3 → Spark → Parquet | Project | intermediate | community | 32.15 |
| Late files | Runbook | intermediate | community | 29.25 |
| Missing partition | Runbook | intermediate | community | 29.25 |
| Apache Spark Documentation | Reference | — | community | 27.7 |
| PostgreSQL Documentation — Tutorial & Reference | Reference | — | community | 27.7 |
| DuckDB and Hugging Face: Querying Datasets Directly | Reference | — | community | 23.55 |
| Importing Data using Java Table Functions | Reference | — | community | 23.55 |
Documented learning objectives
- Identify the five classes of silent failure in a Spark transform
- Add pre/post row-count and null assertions to a job
- Recognise join fan-out caused by non-unique keys
- Read task-level duration histograms to confirm skew
- Choose between AQE skew join, salting and broadcast
- Reproduce skew locally with a bounded dataset
- Scaffold a PySpark project with the template
- Arrange transforms and offline test fixtures