Pattern community intermediate
Compaction & File-Size Management
High-frequency writes shatter a table into thousands of tiny files and queries crawl.
At a glance
| What | High-frequency writes shatter a table into thousands of tiny files and queries crawl. |
|---|---|
| When | Streaming/micro-batch writes to a table; Many small appends per hour |
| Where | Lakehouse |
Problem
High-frequency writes shatter a table into thousands of tiny files and queries crawl.
Solution
Separate the write path from the layout path. writes ─▶ many small files ─▶ [scheduled compaction] ─▶ 128-256 MB files Compact on a schedule (or auto-compact), targeting a file size that matches your read pattern. Rewrite manifests when metadata grows. Do it outside the ingestion window so compaction and writers don't contend.
When not to use
- Low-frequency bulk writes that already produce large files
Security scan
Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.
Context
A lakehouse table written by micro-batches or streaming.
Forces
- Write latency matters
- Read performance matters
- Metadata scanning is expensive too
Tradeoffs
| Choose this when | Costs & caveats |
|---|---|
| fast reads without slowing writes | compaction costs compute and can conflict with concurrent writers |
Diagram
micro-batches ─▶ ⬚⬚⬚⬚⬚⬚⬚⬚ ─▶ [compaction] ─▶ ▭▭▭ (128-256 MB)
Registry stats
| Verification | community · Sep 22, 2026 |
|---|---|
| Completeness | 75 |
| Trust multiplier | ×0.5 |
| Composite | 28.75 · adequate |
| Freshness | 32 days since last source check |
| Difficulty | intermediate |
| Version | 1.1.0 |
| License | CC-BY-4.0 |
| Security flags | 3 |
Did this solve your problem?
Compaction & File-Size Management