Format-driven performance incidents — small scans and wrong layouts

At a glance

Learning objectives

• Map a too-big scan to its layout cause — statistics, row groups, encodings
• Explain how pruning fails when statistics are absent
• Use the lifecycle's cost undercurrent to size the blast radius

What you need

• Lesson 8's read path
• The Parquet docs and format docs bookmarked

Steps

1. Name the incident class — a scan that reads everything for one column; the format documentation is the ground truth for why columnar files scan less data and how row-group pruning works, so start there when a scan does not. (from ref-parquet-format )
2. Check statistics first — understanding statistics and row-group sizing is the practical step that turns use Parquet into and lay out the file like this. (from ref-parquet-docs )
3. Recite the pruning chain — the predicate-pushdown behavior that columnar readers rely on runs on statistics, so a layout without statistics leaves the reader nothing to prune with. (from ref-parquet-format )
4. Treat layout as the fix — row groups and statistics are what turn use Parquet into and lay it out like this, not a bigger cluster. (from ref-parquet-docs )
5. Size the blast radius with the lifecycle — cost is one of the undercurrents of the data engineering lifecycle, alongside security and governance. (from ref-fundamentals-de )
6. Run the review in shared vocabulary — config, logs and process design are where engineers reach for the canonical twelve-factor terms, and a defaulted encoding is a config decision as much as a format one. (from ref-12factor )

Check yourself

• A scan reads every row group despite a filter — you check row-group statistics before scaling anything.
• You can explain why row-group sizing is a performance decision — pruning works per row group on its statistics.

Worked example

Claim (format docs): the ground truth for why columnar files scan less
data and how row-group pruning works.
Leverage (Parquet docs): understanding statistics and row-group sizing
is the practical step that turns "use Parquet" into "and lay out the
file like this".

Sourced from

• Apache Parquet Format Documentation
• Apache Parquet Documentation & Spec
• Fundamentals of Data Engineering (O'Reilly)
• The Twelve-Factor App

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• No statistics, no pruning — the predicate-pushdown behavior columnar readers rely on runs on statistics.
• Use Parquet is not a layout — statistics and row-group sizing are the practical step that turns it into one.
• Default encodings are decisions — a wrong one shows up as a slow scan, and the format docs are the citable source.

Production considerations

• Audit layouts for statistics and row-group sizing — the docs name them as the practical levers.
• Track cost as a lifecycle undercurrent, not an afterthought.

Diagram

format incidents
   |
   |- symptom: a scan reads everything for one column
   |
   |- check 1: row-group statistics present? -> no -> pushdown cannot prune
   |- check 2: row groups sized with intent? -> huge groups -> little pruning
   |- check 3: encodings defaulted? -> format docs are the citable source
   |
    --> fix the layout, not the cluster; cost is a lifecycle undercurrent

Registry stats