ACID and concurrent writes — why compaction and ingestion collide

At a glance

Learning objectives

• Say what ACID transactions on object storage guarantee concurrent writers
• Schedule compaction outside the ingestion window and explain the contention
• Handle a contending OPTIMIZE — compaction is additive, so halt it, no rollback

What you need

• A Delta or Iceberg table with real ingestion traffic
• An ingestion window you can schedule around
• [object Object]

Steps

1. Ground truth first: Delta's official docs cover ACID transactions on object storage, time travel, OPTIMIZE and Z-ORDER clustering, Delta Live Tables, and Unity Catalog integration. (from ref-delta-docs )
2. The classic conflict: do not run compaction during peak ingestion — it conflicts with concurrent writers. (from skill-iceberg-maintenance )
3. The schedule rule: compact after the ingestion window, never during it. (from runbook-small-files-explosion )
4. The pattern behind the rule: separate the write path from the layout path — writes make many small files, scheduled compaction makes 128-256 MB files — and do it outside the ingestion window so compaction and writers don't contend. (from pattern-small-files-compaction )
5. When contention happens anyway: compaction is additive, so there is nothing to roll back — halt any concurrent OPTIMIZE that contends with the ingestion window. (from runbook-small-files-explosion )
6. Guardrail it: Lakehouse Guard is a Delta/Iceberg table maintenance advisor plus a guard that blocks compaction or snapshot expiry during peak ingestion windows. (from plugin-lakehouse-guard )

Check yourself

• You can point at the ingestion window on a calendar and say when compaction may run.
• A compaction plan issued during peak ingestion is blocked by the guard instead of executing.
• You can explain why halting a contending OPTIMIZE needs no rollback.

Worked example

Do not run compaction during peak ingestion; it conflicts with
concurrent writers.

writes ─▶ many small files ─▶ [scheduled compaction] ─▶ 128-256 MB files
Compact on a schedule (or auto-compact), targeting a file size that
matches your read pattern. Do it outside the ingestion window so
compaction and writers don't contend.

Sourced from

• Delta Lake Documentation
• Iceberg Table Maintenance
• Small files explosion
• Compaction & File-Size Management
• Lakehouse Guard

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Compaction during peak ingestion conflicts with concurrent writers — schedule it after the window.
• ZORDER rewrites the whole table — only worth it on a real query predicate.
• Compaction is additive: halt the contending OPTIMIZE, do not attempt a rollback.

Production considerations

• Separate the write path from the layout path: let writers be small and fast, and let scheduled compaction own file size.
• Enforce the window with a guardrail (Lakehouse Guard) rather than a calendar reminder.

Diagram

acid + concurrent writes
   |
   |- writes --> many small files --> peak window: writers only
   |
   |- after the window --> [compaction] --> 128-256 MB files
   |
   |- contending OPTIMIZE? --> halt it (additive, no rollback)
   |
   |- guard --> blocks compaction/expiry during peak ingestion

Registry stats