AWS Track
The most common data platform. S3 → Glue → Athena → Redshift/EMR.
Learning roadmap
The Data Engineering Roadmap
13 stages from Foundations to Agentic Data Engineering. Each has an exit gate you can prove and a deliverable artifact.
Phase 1 — Foundations
Storage, compute, and the primitives everything else builds on.
- 01 Foundations How data systems actually work — storage, compute, encoding, and the vocabulary you'll use for the next decade. 10 guides & tools · 6 further reading Gate You can describe a data system's lifecycle and where cost and latency come from. · Deliverable: Fundamentals of Data Engineering (O'Reilly)
You can, after this stage
- explain the data lifecycle
- read an architecture diagram
- reason about storage vs compute
- 02 Python + SQL The two languages everything else is built from. Fluency here is the single highest-leverage investment. 15 guides & tools · 4 further reading Gate You can move a dataset from an API into a table with tested code. · Deliverable: dbt → warehouse → tests → docs
You can, after this stage
- write set-based SQL
- write testable Python
- move data between the two
- 03 Data Processing Batch engines, file formats, and the mechanics of turning raw bytes into queryable tables. 22 guides & tools · 4 further reading
You can, after this stage
- build a batch pipeline
- choose a file format
- control partitioning
Storage
Databases
OLTP systems, indexes, transactions, replication and how operational data is born.
4 resources
Processing
Spark / PySpark
Distributed dataframes: lazy DAGs, shuffles, skew, partitioning and physical plans.
14 resources
Processing
Batch Processing
Incremental vs full reload, watermarks, idempotent writes and reprocessing strategy.
6 resources
Processing
APIs & Ingestion
Pulling from REST/GraphQL APIs: pagination, rate limits, idempotency keys and incremental syncs.
1 resource
Processing
File & Partner Ingestion
Landing CSV/XML/EDI drops safely: manifesting, validation, late and duplicate files.
1 resource
Phase 2 — Platforms
Warehouses, lakehouses, streaming backbones, and the cloud they run on.
- 04 Warehousing Modelling, transformation and serving — dimensional design, dbt, and warehouse-native patterns. 23 guides & tools · 2 further reading
You can, after this stage
- model a star schema
- write dbt tests
- serve a metric reliably
- 05 Lakehouse Open table formats that give object storage ACID, time travel and schema evolution. 16 guides & tools · 4 further reading Gate You can run and maintain an open table format at scale. · Deliverable: Iceberg Table Maintenance
You can, after this stage
- choose Delta vs Iceberg vs Hudi
- run maintenance
- reason about metadata
- 07 Streaming Event-driven ingestion, log-based CDC, windowing, and exactly-once semantics under load. 23 guides & tools · 4 further reading Gate You can run a streaming pipeline that survives replay. · Deliverable: Kafka → Spark Streaming → Delta
You can, after this stage
- design an event schema
- consume at scale
- recover consumer lag
Processing
Streaming
Event time vs processing time, windowing, watermarks, delivery guarantees and state.
8 resources
Processing
Kafka
Topics, partitions, consumer groups, offsets, retention and lag diagnosis.
13 resources
Processing
CDC
Change data capture: log-based vs query-based, ordering, deletes and merge strategies.
6 resources
- 08 Cloud Provider primitives — object storage, managed compute, IAM, and the platform-specific data stack. 22 guides & tools · 7 further reading
You can, after this stage
- provision least-privilege access
- pick managed vs self-hosted
- control spend
Cloud
AWS
S3, Glue, Athena, Kinesis, Redshift, EMR and IAM — the most common data platform in the world.
19 resources
Cloud
GCP
BigQuery, Cloud Storage, Dataflow, Pub/Sub and Dataproc.
2 resources
Cloud
Databricks
Delta Lake, Unity Catalog, jobs, DLT pipelines and the lakehouse platform.
2 resources
Cloud
Snowflake
Virtual warehouses, micro-partitions, clustering, streams/tasks and Snowpark.
3 resources
Platform
Infrastructure / Docker / Terraform
Containers, images, IaC and reproducible environments for pipelines and platforms.
1 resource
Platform
CI/CD
Testing, packaging and deploying data code — including data-aware CI and safe promotion.
2 resources
Phase 3 — Trust & Operations
Make data reliable: quality checks, observability, orchestration, reliability.
- 06 Orchestration Scheduling, dependencies, retries, backfills and idempotency across many moving parts. 18 guides & tools · 1 further reading Gate You can schedule and recover a multi-step pipeline. · Deliverable: dbt → warehouse → tests → docs
You can, after this stage
- build an idempotent DAG
- plan a safe backfill
- handle late data
- 09 Data Quality Contracts, expectations, quarantine and the tests that keep a table trustworthy on Monday morning. 25 guides & tools · 2 further reading
You can, after this stage
- express a quality contract
- quarantine bad rows
- gate a deploy on tests
- 10 Observability Knowing what a pipeline did without reading its code — metrics, freshness, lineage and cataloguing. 15 guides & tools · 5 further reading Gate You can tell what a pipeline did without reading its code. · Deliverable: OpenLineage Documentation
You can, after this stage
- detect drift automatically
- trace a column to source
- catalogue an asset
Quality
Data Observability
Freshness, volume, schema and distribution monitoring with automated anomaly detection.
10 resources
Quality
Data Lineage
Column-level provenance — what broke, what depends on it and where to fix it.
6 resources
Quality
Data Catalog
Discoverability: technical and business metadata, ownership, glossary and search.
4 resources
- 11 Reliability Operating data systems for years — SLOs, incident response, cost control and safe change. 29 guides & tools · 2 further reading
You can, after this stage
- write a runbook
- respond to an incident
- prevent a repeat failure
Operations
Security
Least privilege, secrets, PII handling, masking and secure data sharing.
3 resources
Operations
Governance
Access policy, retention, residency, audit and stewardship at organisational scale.
4 resources
Operations
Cost Engineering
Unit economics of data — attribution, reducing shuffle/scan, and stopping 4x regressions.
11 resources
Operations
Reliability
SLOs, error budgets, retries vs idempotency, graceful degradation and incident review.
6 resources
Quality
Testing
Unit, integration and data tests; fixtures, contracts and CI gates for pipelines.
1 resource
Operations
Production Operations
Day-two work: monitoring, on-call, backfills, deployments and capacity.
6 resources
Phase 4 — Advanced & Agentic
System design judgment and AI agents as first-class data workers.
- 12 System Design Choosing architectures under real constraints and defending the tradeoffs in an interview. 10 guides & tools · 5 further reading Gate You can whiteboard and defend a data architecture. · Deliverable: Designing Data-Intensive Applications (O'Reilly)
You can, after this stage
- size a design
- articulate tradeoffs
- whiteboard a pipeline
- 13 Agentic Data Engineering Using coding agents as first-class data engineers — skills, MCP servers, hooks and guardrails. 17 guides & tools · 9 further reading
You can, after this stage
- install a skill
- wire an MCP server
- build a review agent
Provider tracks
Platform-specific paths
A focused sequence per provider — combine any track with the main roadmap.
Azure Track
ADLS → Data Factory → Synapse/Fabric with Entra ID governance.
GCP Track
GCS → Dataflow → BigQuery → Pub/Sub.
Databricks Track
The lakehouse platform — Delta, Unity Catalog, jobs and DLT.
Snowflake Track
Micro-partitions, clustering, streams/tasks and Snowpark.