Data Structures & Formats

Hold the row-versus-column mental model

Before you pick a file format, understand why analytics engines read columnar files at all: row groups carrying min/max statistics let a reader prune most of the file before decoding it, and columnar layout means a scan of three columns never touches the other twenty.

Open the Parquet file — row groups, pages, encodings

A Parquet file is not a black box — it is row groups of column chunks made of pages, with statistics that readers use for predicate pushdown.

Arrow — columnar memory and zero-copy interchange

• Explain Arrow's role as the columnar in-memory format and interchange substrate
• Name the pieces — language bindings, IPC and Flight protocols, zero-copy interchange

Avro and schema evolution for event payloads

• Name Avro's four moving parts — schema declaration, evolution rules, encoding, container format
• Explain why schema evolution rules belong to the schema registry

Choosing file formats — Parquet, Avro, and Arrow's role

The three formats in this series are not competitors — they answer different questions.

Compression and encodings — the tradeoffs behind the file

Encodings and compression codecs are where file size and scan speed are actually decided — Parquet applies encodings and codecs per column chunk with statistics on top, while Avro carries schema plus encoding for event payloads.

Format-driven performance incidents — small scans and wrong layouts

When a scan reads the whole lake for one column, the cause is usually the file layout, not the engine — missing statistics, row groups sized without intent, encodings chosen by default.

Reading data files — from bytes to columnar memory

A read is a pipeline in miniature — the reader consults statistics, prunes row groups, decodes only the columns you asked for, and hands you an in-memory columnar structure.

Apache Arrow Documentation

Official Arrow docs: the columnar in-memory format, its language bindings, IPC and Flight protocols, and zero-copy interchange between systems. Use it for exact memory-layout and type-system semantics behind every "columnar" claim in the registry.

Apache Parquet Documentation & Spec

Apache Parquet documentation and specification — columnar encoding, row groups, statistics, predicate pushdown, and how to structure files for efficient reads. Understanding statistics and row-group sizing is the practical step that turns "use Parquet" into "and lay out the file like this"; pairs with small-files and lambda-vs-kappa guides.

Apache Parquet Format Documentation

The Parquet file format documentation: page structure, encodings, compression codecs, statistics and the predicate-pushdown behavior that columnar readers rely on. The ground truth for why columnar files scan less data and how row-group pruning works.