Schema evolution and time travel — change the table, keep the past

At a glance

Learning objectives

• Use schema and partition evolution instead of recreate-and-copy
• Query the history metadata table and time travel within the retention window
• Expire snapshots without breaking time travel — keep one full snapshot

What you need

• A Delta or Iceberg table with a schema you are tempted to change
• Metadata-table access (files, history)
• An ingestion window expiry can be scheduled around

Steps

1. Evolution is a spec feature: Iceberg's documentation covers schema and partition evolution alongside the table format, snapshots and manifests, and hidden partitioning. (from ref-iceberg-docs )
2. Delta's side of the bargain: its official docs cover ACID transactions on object storage, time travel, OPTIMIZE and Z-ORDER clustering, Delta Live Tables, and Unity Catalog integration. (from ref-delta-docs )
3. See the past with SQL: for Iceberg/Delta, query the metadata tables (files, history). (from runbook-small-files-explosion )
4. Bound the past safely: expire_snapshots older than the retention window — but keep at least one full snapshot for time travel. (from skill-iceberg-maintenance )
5. Evolution without loss starts at the bottom zone: raw/ — immutable, exactly as received; never edited; schema-on-read. (from pattern-medallion-zones )
6. Reprocessing needs the original bytes — that force is why raw is immutable and why an evolved table can always be rebuilt from its source. (from pattern-medallion-zones )
7. Schedule expiry like compaction: Lakehouse Guard is a guard that blocks compaction or snapshot expiry during peak ingestion windows. (from plugin-lakehouse-guard )

Check yourself

• You can time travel to a snapshot inside the retention window after expiry ran.
• A schema change on curated/ required no rewrite of raw/ — the original bytes were never edited.
• The history metadata table answers what changed and when without leaving SQL.

Worked example

`expire_snapshots` older than the retention window — but keep at least
one full snapshot for time travel.

raw/       — immutable, exactly as received; never edited; schema-on-read.

Sourced from

• Apache Iceberg Documentation
• Delta Lake Documentation
• Small files explosion
• Iceberg Table Maintenance
• Medallion Zones (raw / curated / serving)
• Lakehouse Guard

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• expire_snapshots must keep at least one full snapshot or time travel breaks.
• Schema and partition evolution are spec features — the format, not a copy job, is the mechanism.
• Snapshot expiry is maintenance with a window: the guard blocks it during peak ingestion.

Production considerations

• Set the snapshot retention window deliberately — it is the knob that trades storage for time-travel depth.
• Keep raw/ immutable and schema-on-read so every evolution has an original to return to.

Diagram

schema evolution & time travel
   |
   |- spec: schema evolution | partition evolution | hidden partitioning
   |
   |- db.tbl.history --> what changed, when
   |
   |- time travel --> snapshot inside the retention window
   |
   |- expire_snapshots --> keep one full snapshot
   |
   raw (immutable, schema-on-read) --> curated (evolved) --> serving

Registry stats