Table format mechanics — snapshots, manifests, and the metadata tables

At a glance

Learning objectives

• Name the layers the Iceberg spec defines — snapshots, manifests, hidden partitioning
• Query the files and history metadata tables with plain SQL
• Explain when manifests need rewriting and why the spec outranks engine docs

What you need

• An Iceberg table you can query (any engine that reads the format)
• Read access to the table's metadata tables (files, history)
• No installs — the spec and the metadata tables are the materials

Steps

1. A table format is a specification: the spec is the ground truth when tools disagree. (from ref-iceberg-docs )
2. The spec defines the layers: the table format, snapshots and manifests, hidden partitioning, schema and partition evolution, and maintenance procedures like compaction and snapshot expiry. (from ref-iceberg-docs )
3. Metadata is queryable SQL: for Iceberg/Delta, query the metadata tables (files, history). (from runbook-small-files-explosion )
4. Start with files: SELECT count(*) files, avg(file_size_in_bytes)/1e6 avg_mb FROM db.tbl.files; — avg_mb < 32 and a growing count means compact. (from skill-iceberg-maintenance )
5. Metadata itself needs maintenance: run rewrite_manifests when the manifest list is huge. (from skill-iceberg-maintenance )
6. One format is not the whole story: Amazon EMR's multi-catalog queries Iceberg, Delta Lake, Hudi, and Hive tables through a single catalog and joins data across AWS accounts without copying it. (from ref-query-across-accounts-and-table-formats-with-multi-catalog-in-am )

Check yourself

• db.tbl.files returns a count and an average file size on your table, and you can say which layer of the spec it reads.
• You can explain what rewrite_manifests fixes and when to run it — when the manifest list is huge.
• You can say why the spec, not any engine's docs, settles a disagreement between two engines over the same table.

Worked example

SELECT count(*) files, avg(file_size_in_bytes)/1e6 avg_mb
FROM db.tbl.files;

1. `rewrite_data_files` with a target size (128-256 MB).
2. `rewrite_manifests` when the manifest list is huge.
3. `expire_snapshots` older than the retention window — but keep at least one
   full snapshot for time travel.
4. `remove_orphan_files` on a schedule, with a safety margin.

Sourced from

• Apache Iceberg Documentation
• Small files explosion
• Iceberg Table Maintenance
• Query across accounts and table formats with multi-catalog in Amazon EMR

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• The spec is the ground truth when tools disagree — do not settle format questions from an engine's blog post.
• A huge manifest list is its own problem: rewrite_manifests, not just data-file compaction.
• Metadata tables (files, history) are queried like tables — no special tooling required.

Production considerations

• Read the spec once end to end: it covers snapshots, manifests, hidden partitioning, evolution, and the maintenance procedures your skills automate.
• Multi-engine shops share tables through catalogs; verify which catalog each engine resolves before comparing query plans.

Diagram

iceberg metadata mechanics
   |
   spec: snapshots | manifests | hidden partitioning | evolution
   |
   |- db.tbl.files  --> count, avg file size --> compact trigger
   |
   |- db.tbl.history --> what happened, when
   |
   |- manifest list huge --> rewrite_manifests
   |
   |- one catalog --> Iceberg + Delta + Hudi + Hive

Registry stats