All categories

How data systems actually work — storage, compute, encoding, and the vocabulary you'll use for the next decade.

• Engineering Foundations Filesystems, encoding, memory vs disk, networks and the operating vocabulary behind every pipeline. 4 resources Core
• Data Structures & Formats Row vs columnar, Parquet vs ORC vs Avro, encoding, compression and predicate pushdown. 12 resources Storage

The two languages everything else is built from. Fluency here is the single highest-leverage investment.

• Python Typing, packaging, virtualenvs and the data libraries (pandas, polars, pyarrow) you'll actually reach for. 1 resource Language
• SQL Set-based thinking, window functions, CTEs and query plans. The highest-ROI skill in the field. 18 resources Language

Batch engines, file formats, and the mechanics of turning raw bytes into queryable tables.

• Databases OLTP systems, indexes, transactions, replication and how operational data is born. 4 resources Storage
• Spark / PySpark Distributed dataframes: lazy DAGs, shuffles, skew, partitioning and physical plans. 14 resources Processing
• Batch Processing Incremental vs full reload, watermarks, idempotent writes and reprocessing strategy. 6 resources Processing
• APIs & Ingestion Pulling from REST/GraphQL APIs: pagination, rate limits, idempotency keys and incremental syncs. 1 resource Processing
• File & Partner Ingestion Landing CSV/XML/EDI drops safely: manifesting, validation, late and duplicate files. 1 resource Processing

Modelling, transformation and serving — dimensional design, dbt, and warehouse-native patterns.

• Data Warehouses Columnar MPP serving layers, dimensional modelling, marts and semantic layers. 5 resources Serving
• dbt Analytics engineering as software: models, refs, tests, docs and incremental materialisations. 20 resources Serving

Open table formats that give object storage ACID, time travel and schema evolution.

• Data Lakes Object-storage data lakes, zones (raw/curated), partitioning strategy and the small-files problem. 2 resources Storage
• Lakehouse Delta Lake, Apache Iceberg and Hudi — ACID tables on object storage with time travel and evolution. 18 resources Storage

Event-driven ingestion, log-based CDC, windowing, and exactly-once semantics under load.

• Streaming Event time vs processing time, windowing, watermarks, delivery guarantees and state. 8 resources Processing
• Kafka Topics, partitions, consumer groups, offsets, retention and lag diagnosis. 13 resources Processing
• CDC Change data capture: log-based vs query-based, ordering, deletes and merge strategies. 6 resources Processing

Provider primitives — object storage, managed compute, IAM, and the platform-specific data stack.

• AWS S3, Glue, Athena, Kinesis, Redshift, EMR and IAM — the most common data platform in the world. 19 resources Cloud
• Azure ADLS, Synapse, Data Factory, Fabric and Entra ID for the Microsoft-shaped data stack. 0 resources Cloud
• GCP BigQuery, Cloud Storage, Dataflow, Pub/Sub and Dataproc. 2 resources Cloud
• Databricks Delta Lake, Unity Catalog, jobs, DLT pipelines and the lakehouse platform. 2 resources Cloud
• Snowflake Virtual warehouses, micro-partitions, clustering, streams/tasks and Snowpark. 3 resources Cloud
• Infrastructure / Docker / Terraform Containers, images, IaC and reproducible environments for pipelines and platforms. 1 resource Platform
• CI/CD Testing, packaging and deploying data code — including data-aware CI and safe promotion. 2 resources Platform

Scheduling, dependencies, retries, backfills and idempotency across many moving parts.

• Airflow / Orchestration DAGs, operators, task state, retries, backfills and the difference between scheduling and dependency management. 19 resources Orchestration

Contracts, expectations, quarantine and the tests that keep a table trustworthy on Monday morning.

• Data Quality Expectations, completeness, uniqueness and quarantine flows that stop bad data reaching consumers. 14 resources Quality
• Data Contracts Versioned producer/consumer agreements: schema, semantics, SLAs and breaking-change policy. 13 resources Quality

Knowing what a pipeline did without reading its code — metrics, freshness, lineage and cataloguing.

• Data Observability Freshness, volume, schema and distribution monitoring with automated anomaly detection. 10 resources Quality
• Data Lineage Column-level provenance — what broke, what depends on it and where to fix it. 6 resources Quality
• Data Catalog Discoverability: technical and business metadata, ownership, glossary and search. 4 resources Quality

Operating data systems for years — SLOs, incident response, cost control and safe change.

• Security Least privilege, secrets, PII handling, masking and secure data sharing. 3 resources Operations
• Governance Access policy, retention, residency, audit and stewardship at organisational scale. 4 resources Operations
• Cost Engineering Unit economics of data — attribution, reducing shuffle/scan, and stopping 4x regressions. 11 resources Operations
• Reliability SLOs, error budgets, retries vs idempotency, graceful degradation and incident review. 6 resources Operations
• Testing Unit, integration and data tests; fixtures, contracts and CI gates for pipelines. 1 resource Quality
• Production Operations Day-two work: monitoring, on-call, backfills, deployments and capacity. 6 resources Operations

Choosing architectures under real constraints and defending the tradeoffs in an interview.

• Data Engineering System Design Sizing, throughput, tradeoffs and communicating an architecture under interview pressure. 15 resources Practice

Using coding agents as first-class data engineers — skills, MCP servers, hooks and guardrails.

• Agentic Data Engineering Skills, MCP servers, agents and hooks that let coding agents do real data engineering safely. 26 resources Practice
• AI / ML Data Infrastructure Feature stores, embeddings, vector indexes, training data pipelines and retrieval infra. 0 resources Practice