Data Engineering System Design

Hold the system design mental model — data systems as tradeoffs

• Frame any data system as a tradeoff among reliability, scalability, and maintainability
• Name the six mechanisms DDIA teaches and map each one to a system you operate

Run the back-of-envelope drill — numbers before architecture

Estimation is the warm-up act of every design conversation: QPS, storage per day, retention, peak factor, all written down before a single box is drawn.

Choose storage engines on tradeoffs — reads, writes, and consistency

Storage choice is where a design's promises become real: the engine you pick decides what replication buys, what partitioning costs, and which consistency semantics your merges get.

Define reliability in numbers — SLIs, SLOs, and error budgets

• Define an SLI and an SLO for a data property like freshness or completeness
• Compute an error budget and state what burning it permits and forbids

Design resilient pipelines — retry, circuit breaker, bulkhead

• Match each pipeline failure shape to its named pattern and its considerations
• Cap retries with backoff and know when a circuit breaker replaces them

Read reference architectures like an engineer — AWS as precedent

• Decompose a reference architecture into ingest, store, process, and serve
• Connect each architectural choice to a Well-Architected pillar

Speak the scaling vocabulary — load balancing, caching, sharding

Scale arguments go wrong when the words do: horizontal versus vertical, cache-aside versus write-through, shard by range versus hash.

Capstone checklist — a design is done when these all hold

Nine lessons compress into one checklist: tradeoffs named, numbers written, storage chosen by mechanism, scaling lever identified, budget defined, failure patterns assigned, precedent cited.

Interview walkthrough — design a data platform end-to-end

The full arc in one 45-minute shape: requirements, back-of-envelope numbers, the storage and scaling argument, reliability as SLIs and budgets, and a deployed precedent to close.

Production gotchas — premature scaling and the missing capacity plan

The two self-inflicted design failures: complexity bought before load requires it, and capacity treated as an afterthought until the pager goes off.

AWS Architecture Center

AWS's official collection of architecture diagrams, reference architectures and best-practice guidance (Well-Architected pillars, serverless data lake patterns). Concrete vendor material for grounding sizing and tradeoff arguments in real deployed patterns.

Azure Cloud Design Patterns

Microsoft's catalogue of cloud design patterns: retry, circuit breaker, compensating transaction, CQRS, event sourcing, bulkhead and more — each with problem/solution and considerations. Vendor-neutral in practice, and directly applicable to pipeline component design.

Designing Data-Intensive Applications (O'Reilly)

The canonical book on the principles behind data systems: reliability, scalability, replication, partitioning, consistency, and batch/stream processing. Martin Kleppmann's DDIA is the single highest-value read for system-design interviews and for understanding why data tools are shaped as they are. Every chapter maps directly to patterns in this registry — replication to CDC, partitioning to lakehouse, consistency to transaction semantics. The definitive textbook for data engineers.

Google SRE Books

Google's Site Reliability Engineering books, free online: SLIs/SLOs/error budgets, incident management, postmortem culture and capacity planning. The vocabulary source for reliability engineering that data platform teams borrow from when they define data SLAs and run incident drills.

The System Design Primer

The widely-used open-source study guide for system design: scaling vocabulary (load balancers, caching, sharding), back-of-envelope estimation, and worked design exercises. Data platform architectures are a special case of the same throughput/consistency tradeoffs it teaches.