Canonical taxonomy
Data engineering taxonomy
38 categorys grouped into 13 roadmap stages. Every resource maps to exactly one category; categories map to stages. This is the spine for the roadmap, the registry and search.
Resource types
15
skills, MCP servers, plugins, agents, hooks, runbooks…
Categories
38
exactly one per resource
Roadmap stages
13
categories group into stages
Technologies
31
first-class references
Stages & categories
The full mapping
Every stage, its categories, and the resource types that live in each — all counts derived from the registry.
Foundations
roadmap| Category | Resources | Types present |
|---|---|---|
| Engineering Foundations Filesystems, encoding, memory vs disk, networks and the operating vocabulary behind every pipeline. | 4 | |
| Data Structures & Formats Row vs columnar, Parquet vs ORC vs Avro, encoding, compression and predicate pushdown. | 12 |
Python + SQL
roadmapData Processing
roadmap| Category | Resources | Types present |
|---|---|---|
| Databases OLTP systems, indexes, transactions, replication and how operational data is born. | 4 | |
| Spark / PySpark Distributed dataframes: lazy DAGs, shuffles, skew, partitioning and physical plans. | 14 | |
| Batch Processing Incremental vs full reload, watermarks, idempotent writes and reprocessing strategy. | 6 | |
| APIs & Ingestion Pulling from REST/GraphQL APIs: pagination, rate limits, idempotency keys and incremental syncs. | 1 | |
| File & Partner Ingestion Landing CSV/XML/EDI drops safely: manifesting, validation, late and duplicate files. | 1 |
Warehousing
roadmap| Category | Resources | Types present |
|---|---|---|
| Data Warehouses Columnar MPP serving layers, dimensional modelling, marts and semantic layers. | 5 | |
| dbt Analytics engineering as software: models, refs, tests, docs and incremental materialisations. | 20 |
Lakehouse
roadmapOrchestration
roadmap| Category | Resources | Types present |
|---|---|---|
| Airflow / Orchestration DAGs, operators, task state, retries, backfills and the difference between scheduling and dependency management. | 19 |
Streaming
roadmap| Category | Resources | Types present |
|---|---|---|
| Streaming Event time vs processing time, windowing, watermarks, delivery guarantees and state. | 8 | |
| Kafka Topics, partitions, consumer groups, offsets, retention and lag diagnosis. | 13 | |
| CDC Change data capture: log-based vs query-based, ordering, deletes and merge strategies. | 6 |
Cloud
roadmap| Category | Resources | Types present |
|---|---|---|
| AWS S3, Glue, Athena, Kinesis, Redshift, EMR and IAM — the most common data platform in the world. | 19 | |
| Azure ADLS, Synapse, Data Factory, Fabric and Entra ID for the Microsoft-shaped data stack. | 0 | |
| GCP BigQuery, Cloud Storage, Dataflow, Pub/Sub and Dataproc. | 2 | |
| Databricks Delta Lake, Unity Catalog, jobs, DLT pipelines and the lakehouse platform. | 2 | |
| Snowflake Virtual warehouses, micro-partitions, clustering, streams/tasks and Snowpark. | 3 | |
| Infrastructure / Docker / Terraform Containers, images, IaC and reproducible environments for pipelines and platforms. | 1 | |
| CI/CD Testing, packaging and deploying data code — including data-aware CI and safe promotion. | 2 |
Data Quality
roadmap| Category | Resources | Types present |
|---|---|---|
| Data Quality Expectations, completeness, uniqueness and quarantine flows that stop bad data reaching consumers. | 14 | |
| Data Contracts Versioned producer/consumer agreements: schema, semantics, SLAs and breaking-change policy. | 13 |
Observability
roadmap| Category | Resources | Types present |
|---|---|---|
| Data Observability Freshness, volume, schema and distribution monitoring with automated anomaly detection. | 10 | |
| Data Lineage Column-level provenance — what broke, what depends on it and where to fix it. | 6 | |
| Data Catalog Discoverability: technical and business metadata, ownership, glossary and search. | 4 |
Reliability
roadmap| Category | Resources | Types present |
|---|---|---|
| Security Least privilege, secrets, PII handling, masking and secure data sharing. | 3 | |
| Governance Access policy, retention, residency, audit and stewardship at organisational scale. | 4 | |
| Cost Engineering Unit economics of data — attribution, reducing shuffle/scan, and stopping 4x regressions. | 11 | |
| Reliability SLOs, error budgets, retries vs idempotency, graceful degradation and incident review. | 6 | |
| Testing Unit, integration and data tests; fixtures, contracts and CI gates for pipelines. | 1 | |
| Production Operations Day-two work: monitoring, on-call, backfills, deployments and capacity. | 6 |
System Design
roadmap| Category | Resources | Types present |
|---|---|---|
| Data Engineering System Design Sizing, throughput, tradeoffs and communicating an architecture under interview pressure. | 15 |
Agentic Data Engineering
roadmap| Category | Resources | Types present |
|---|---|---|
| Agentic Data Engineering Skills, MCP servers, agents and hooks that let coding agents do real data engineering safely. | 26 | |
| AI / ML Data Infrastructure Feature stores, embeddings, vector indexes, training data pipelines and retrieval infra. | 0 | |
Technologies
First-class references
Technology references used by resources across the registry, with derived usage counts.
| Technology | Domain | Resources | Reference |
|---|---|---|---|
| Python | cat-python | 5 | docs |
| SQL | cat-sql | 3 | — |
| PostgreSQL | cat-databases | 4 | docs |
| DuckDB | cat-sql | 5 | docs |
| Apache Arrow | cat-data-formats | 0 | docs |
| Apache Parquet | cat-data-formats | 4 | docs |
| Apache Avro | cat-data-formats | 2 | docs |
| Apache Spark | cat-spark | 13 | docs |
| PySpark | cat-spark | 3 | docs |
| Apache Kafka | cat-kafka | 5 | docs |
| Apache Flink | cat-streaming | 0 | docs |
| Apache Airflow | cat-airflow | 4 | docs |
| Dagster | cat-airflow | 0 | docs |
| dbt | cat-dbt | 9 | docs |
| Delta Lake | cat-lakehouse | 5 | docs |
| Apache Iceberg | cat-lakehouse | 3 | docs |
| Apache Hudi | cat-lakehouse | 0 | docs |
| Debezium | cat-cdc | 2 | docs |
| Great Expectations | cat-data-quality | 0 | docs |
| Soda Core | cat-data-quality | 0 | docs |
| Docker | cat-infrastructure | 3 | docs |
| Terraform | cat-infrastructure | 2 | docs |
| Databricks | cat-databricks | 4 | docs |
| Snowflake | cat-snowflake | 6 | docs |
| BigQuery | cat-gcp | 3 | docs |
| Amazon S3 | cat-aws | 5 | docs |
| AWS Glue | cat-aws | 4 | docs |
| Amazon Athena | cat-aws | 3 | docs |
| OpenLineage | cat-data-lineage | 1 | docs |
| DataHub | cat-data-catalog | 2 | docs |
| AWS | cat-aws | 1 | docs |