Core

Engineering Foundations

Filesystems, encoding, memory vs disk, networks and the operating vocabulary behind every pipeline.

4 resources 2 types

Inside this category

It currently holds 1 resource: 1 reference. Mapped to roadmap stage: Foundations. Verification mix: 1 community — check each entry's evidence before adopting.

Where to start: Fundamentals of Data Engineering (O'Reilly) (33.86)

Tutorial 34 adequate

The twelve-factor vocabulary for pipeline jobs

Data pipelines run as jobs on shared infrastructure, so the twelve-factor principles apply to them — config in the environment, backing services as attached resources, logs as event streams, disposability, and dev/prod parity.

Details

beginner tutorialtwelve-factor community

Tutorial 34 adequate

Capstone — the foundations checklist

Ten lessons compress into one checklist — place every decision on the lifecycle, lay out analytics files with row groups and statistics, expect Arrow at the boundaries, evolve event payloads with Avro's rules, and run jobs twelve-factor style.

Details

intermediate tutorialfoundations community

Reference 27 adequate

Fundamentals of Data Engineering (O'Reilly)

A lifecycle view of data engineering — generation, storage, ingestion, transformation, and serving — with the "undercurrents" of security, governance, and cost. Joe Reis and Matt Housley provide a modern, practitioner-oriented map of the field. The best overview for someone moving into data engineering; pairs perfectly with this platform's roadmap. Covers the full lifecycle from ingestion through serving, with practical patterns for each stage.

Details

bookfoundations community

Reference 28 adequate

The Twelve-Factor App

The classic twelve principles for building services that run portably: config in the environment, backing services as attached resources, logs as event streams, disposability and dev/prod parity. Data pipelines inherit these ideas whenever they run as jobs on shared infrastructure.

Details

methodologyconfig community