Capstone — the standing lakehouse checklist

At a glance

Learning objectives

• Run the full standing checklist on one table you own
• Turn the four-part maintenance plan into a scheduled job with alerts
• Apply the escalation line — page only on a freshness-SLA threat

What you need

• One production lakehouse table you own
• [object Object]
• [object Object]

Steps

1. Check the zones: raw/ is immutable, exactly as received, never edited, schema-on-read; curated/ is cleaned, typed, deduplicated, quality-gated; serving/ is SLA-bearing — and each zone has a different retention, quality bar and owner, with promotion between zones a pipeline step with a gate, not a copy. (from pattern-medallion-zones )
2. Check the format decision: Delta's docs are the definitive source for Delta features and the maintenance operations the lakehouse skills reference — compare with a Delta vs Iceberg decision guide before committing to a format. (from ref-delta-docs )
3. Check the monitors: file count, avg file size, planning time. (from runbook-small-files-explosion )
4. Check the maintenance plan: rewrite_data_files with a target size (128-256 MB), rewrite_manifests when the manifest list is huge, expire_snapshots within retention keeping one full snapshot for time travel, remove_orphan_files on a schedule with a safety margin. (from skill-iceberg-maintenance )
5. Check the automation: enable Delta auto-compaction or schedule Iceberg rewrite_data_files; target 128-256 MB files and alert on file-count growth. (from runbook-small-files-explosion )
6. Check the guardrails: /table-health for a status read, /plan-compaction for the maintenance plan, /expire-snapshots for the expiry plan — plans first, not ad hoc runs. (from plugin-lakehouse-guard )
7. Check the escalation line: no page — schedule compaction; escalate only if planning time threatens a critical dashboard's freshness SLA. (from runbook-small-files-explosion )

Check yourself

• Every zone has a named owner, a quality bar, and a gate — and curated is not a dumping ground.
• The three monitors (file count, avg file size, planning time) have alerts, and 128-256 MB is the recorded target.
• A /plan-compaction during peak ingestion is blocked; the same plan after the window runs.

Worked example

file count, avg file size, planning time

1. `rewrite_data_files` with a target size (128-256 MB).
2. `rewrite_manifests` when the manifest list is huge.
3. `expire_snapshots` older than the retention window — but keep at least one
   full snapshot for time travel.
4. `remove_orphan_files` on a schedule, with a safety margin.

/table-health
/plan-compaction
/expire-snapshots

Sourced from

• Medallion Zones (raw / curated / serving)
• Delta Lake Documentation
• Small files explosion
• Iceberg Table Maintenance
• Lakehouse Guard

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Compaction and snapshot expiry never run during peak ingestion — the guard blocks them, and so should you.
• Curated becomes a dumping ground without an enforced gate — promotion is a pipeline step, not a copy.
• Escalate only when planning time threatens a critical dashboard's freshness SLA — otherwise schedule, do not page.

Production considerations

• The four-part plan (data files, manifests, snapshots, orphans) is one scheduled job with a safety margin — never four ad hoc commands.
• Alert on file-count growth and target 128-256 MB files; those two numbers catch most lakehouse decay early.

Diagram

lakehouse capstone
   |
   |- zones: raw --> curated --> serving (gated promotion)
   |
   |- format: Delta vs Iceberg, chosen deliberately
   |
   |- monitors: file count | avg file size | planning time
   |
   |- maintenance: data files | manifests | snapshots | orphans
   |
   |- guard: /table-health /plan-compaction /expire-snapshots
   |
   |- escalation: page only on a freshness-SLA threat

Registry stats