Compression and encodings — the tradeoffs behind the file

At a glance

Learning objectives

• Distinguish encodings from compression codecs in a columnar file
• Explain how statistics sit on top of encoded, compressed data to enable pushdown
• Compare Parquet's columnar encoding with Avro's encoding for event payloads

What you need

• Lesson 2's Parquet anatomy
• The Parquet format docs and the Avro docs bookmarked

Steps

1. Get the layering from the spec — the Parquet format documentation covers page structure, encodings, compression codecs, statistics, and the predicate-pushdown behavior that columnar readers rely on. (from ref-parquet-format )
2. Treat encoding and compression as file-layout questions — when file-layout questions come up (row groups, min/max statistics, compression), the format docs are the citable source. (from ref-parquet-format )
3. Connect encoding back to layout — the Parquet documentation covers columnar encoding, row groups, statistics, predicate pushdown, and how to structure files for efficient reads. (from ref-parquet-docs )
4. See the event-side counterpart — Avro's official documentation covers schema declaration, the schema-registry evolution rules, encoding, and the container format widely used for event and stream payloads. (from ref-avro-docs )
5. Keep the Arrow boundary in mind — Arrow's docs describe the columnar in-memory format with zero-copy interchange, the layer these on-disk encodings feed into and out of. (from ref-arrow-docs )

Check yourself

• You can name the layers a Parquet reader passes through — pages with encodings, compression codecs, then statistics for pushdown.
• Asked where encoding lives for an event payload, you point at Avro's encoding and container format rather than the columnar file.

Worked example

Parquet format docs: page structure, encodings, compression codecs,
statistics and the predicate-pushdown behavior that columnar readers
rely on.

Avro docs: schema declaration, schema-registry evolution rules, encoding
and the container format widely used for event/stream payloads and
Kafka wire formats.

Official Arrow docs: the columnar in-memory format, its language
bindings, IPC and Flight protocols, and zero-copy interchange between
systems.

Sourced from

• Apache Parquet Format Documentation
• Apache Parquet Documentation & Spec
• Apache Avro Documentation
• Apache Arrow Documentation

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Encodings, codecs, and statistics are three different layers — the format documentation lists page structure, encodings, compression codecs, statistics, and pushdown separately.
• Columnar encodings serve scans — Avro's encoding serves event and stream payloads — the same word, different jobs.

Production considerations

• Pick encodings and codecs as a layout decision — structure files for efficient reads, per the Parquet documentation.
• For event payloads, the container format and evolution rules come from the Avro docs, not from the columnar spec.

Diagram

compression + encodings
   |
   |- parquet column chunk: pages
   |      encodings -> compression codecs -> statistics (min/max)
   |      reader: statistics -> predicate pushdown -> decode less
   |
   |- avro payload: schema declaration -> encoding -> container file
   |      the format for event / stream payloads and Kafka wire formats
   |
   |- arrow: columnar in-memory, zero-copy interchange (no file layout)
   |
    --> on-disk columnar (parquet) | event container (avro) | in-memory (arrow)

Registry stats