Series · roadmap stage 5/13
The Lakehouse Series
Ten lessons in order, from the mental model to a capstone checklist. Every lesson is structured by us and grounded in a real, cited source from the registry — no invented commands.
10 of 10 lessons published
- Lesson 1 The lakehouse mental model — lake vs warehouse vs table formats vs zones
Build the map before touching a table: object storage is files with no structure for trust or ownership, a table format (Iceberg or Delta) adds snapshots, manifests, hidden partitioning, ACID, and evolution, and medallion zones make trust l
- Lesson 2 Your first lakehouse table — create, query, measure, compact
Run the whole loop in one sitting on a scratch table: create and query a first Delta or Iceberg table from the official docs, read its health from the files metadata table with one SQL query, run a first OPTIMIZE or rewrite_data_files, and
- Lesson 3 Table format mechanics — snapshots, manifests, and the metadata tables
Open the hood on an Iceberg table: the spec's layers (snapshots, manifests, hidden partitioning), the files and history metadata tables you query with plain SQL, and the manifest list as a maintenance target in its own right. One catalog th
- Lesson 4 ACID and concurrent writes — why compaction and ingestion collide
What ACID on object storage actually guards, and where it stops: Delta commits transactions on files, but compaction is also a writer, and the two meet in the ingestion window. Learn the schedule rule, the write-path/layout-path separation,
- Lesson 5 Run the Iceberg maintenance routine
Work an Iceberg table through the maintenance skill's plan: read the files metadata table to size the problem, rewrite data files to a 128-256 MB target, rewrite manifests when the manifest list is huge, expire snapshots without breaking ti
- Lesson 6 Schema evolution and time travel — change the table, keep the past
Change a table without breaking its readers or its history: the Iceberg spec's schema and partition evolution, Delta time travel, the history metadata table you query to see what happened, and the snapshot-expiry rule that keeps time travel
- Lesson 7 Check lakehouse table health with Lakehouse Guard
Use the Lakehouse Guard plugin's three commands on a Delta/Iceberg table: /table-health for a status read, /plan-compaction for a maintenance plan, and /expire-snapshots for a safe expiry plan. The guard exists to block compaction or snapsh
- Lesson 8 Decide when a Delta table needs OPTIMIZE or Z-ORDER
Apply the Delta Optimization Advisor's decision rules to one table: read file stats for the OPTIMIZE trigger, pick Z-ORDER columns from actual query predicates, weigh liquid clustering for high-cardinality write-heavy tables, and prove the
- Lesson 9 Diagnose a small-files explosion
Follow the small-files runbook from symptom to prevention: confirm the tens-of-thousands-of-tiny-files signature, query the metadata tables for the average file size, compact to a 128-256 MB target, coarsen an over-partitioned layout, and s
- Lesson 10 Capstone — the standing lakehouse checklist
Close the series by running the full checklist on one table you own: zones with gates, a deliberately chosen format, the three standing monitors, the four-part maintenance plan as a scheduled job, guardrail-enforced windows, and the escalat