Reading data files — from bytes to columnar memory

At a glance

Learning objectives

• Describe the read path — statistics, pruning, column decoding, in-memory handoff
• Use predicate-pushdown vocabulary when a scan reads more than the query asked
• Place the read path on the lifecycle between storage and transformation

What you need

• Lesson 2's file anatomy
• A Parquet file you are allowed to read — the concepts hold without one

Steps

1. Start from the lifecycle — after generation and storage comes ingestion, then transformation; reading a file sits at that boundary, and the fundamentals book gives practical patterns for each stage. (from ref-fundamentals-de )
2. Follow the reader's own logic — the format documentation describes the statistics and predicate-pushdown behavior that columnar readers rely on. (from ref-parquet-format )
3. Name what pruning skips — row-group pruning works on per-row-group statistics, the mechanism the Parquet documentation and specification covers under row groups and statistics. (from ref-parquet-docs )
4. Know what lands in memory — after decoding, the columns live in the columnar in-memory format; the Arrow docs cover that format, its language bindings, IPC and Flight protocols, and zero-copy interchange. (from ref-arrow-docs )
5. Keep the map of the field nearby — the fundamentals book is the practitioner-oriented map for someone moving into data engineering, and the read path is one of its stages. (from ref-fundamentals-de )

Check yourself

• You can narrate a scan — statistics first, pruned row groups next, only the requested columns decoded, handed over in columnar memory.
• A scan reads a whole file for two columns — you ask what the row-group statistics said before blaming the engine.

Worked example

statistics + predicate-pushdown behavior that columnar readers rely on
-> row-group pruning (ground truth per the Parquet format docs)
-> decode the requested columns
-> columnar in-memory format, zero-copy interchange (Arrow docs)

Sourced from

• Fundamentals of Data Engineering (O'Reilly)
• Apache Parquet Format Documentation
• Apache Parquet Documentation & Spec
• Apache Arrow Documentation

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Predicate pushdown depends on statistics — without them the reader has nothing to prune with; the format docs name statistics and pushdown as a pair.
• Reading is not one step — file format, then in-memory format; the Arrow docs describe the second layer.

Production considerations

• Structure files for efficient reads — the Parquet documentation names statistics, row groups, and predicate pushdown as the levers.
• Expect the in-memory shape to be Arrow — the substrate most modern data tools exchange data through.

Diagram

the read path
   |
   |- query: two columns, one predicate
   |
   |- footer: row-group statistics (min/max)
   |
   |- predicate pushdown --> prunes row groups that cannot match
   |
   |- decode only the requested columns (pages, encodings, codecs)
   |
   |- handoff: columnar in-memory format (Arrow-shaped)
   |
    --> fewer bytes decoded, zero-copy between systems

Registry stats