Build
Architecture patterns
Named, reusable solutions to recurring data engineering problems — each with context, forces, trade-offs, and when not to use it.
dbt as Models + Tests + Docs
Transformation logic lives in untested, undocumented SQL scattered across scripts.
Details
Idempotent Partition Overwrite
Retries and backfills duplicate or corrupt data because writes are appends.
Details
Medallion Zones (raw / curated / serving)
Object storage gives you a place to put files but no structure for trust or ownership.
Details
Agent-in-the-Loop Review
Coding agents can do real data-engineering work, but can also ship a schema change or drop a table.
Details
Compaction & File-Size Management
High-frequency writes shatter a table into thousands of tiny files and queries crawl.
Details
Contracts as a Quality Gate
Producers change schemas and semantics with no notice; consumers break in production.
Details
Cost Guardrails
A working pipeline silently becomes expensive and nobody notices until the invoice.
Details
Dimensional Model (star schema)
Analysts re-derive the same joins and disagree on metric definitions.
Details
Incremental Batch with a Watermark
Full reloads are too slow and too expensive once a table grows past millions of rows.
Details
Quarantine then Publish
One bad batch of rows can poison a whole table and every downstream metric.
Details
Data Vault (hubs, links, satellites)
Integrating many changing sources into a warehouse without constant rework.
Details
Lambda vs Kappa (batch + stream)
Streaming is fast but hard; batch is simple but slow; you need both speed and correctness.
Details
Postgres CDC to Lakehouse
Batch extraction from an OLTP database loads it and still misses deletes and mid-day changes.
Details
Safe Backfill
Backfills corrupt production or take down the daily run.
Details
Streaming Dedup and Upsert
Streaming sinks either duplicate on replay or lose rows on failure.