Choose storage engines on tradeoffs — reads, writes, and consistency

At a glance

Learning objectives

• Argue a storage choice from replication, partitioning, and consistency needs
• Explain what each consistency semantic means for a merge-based pipeline
• Recognize read/write shape patterns (CQRS, event sourcing) in vendor storage

What you need

• Lesson 1's six-axis vocabulary
• One storage decision currently on your plate

Steps

1. Re-anchor on the mechanisms — DDIA's core chapters are replication, partitioning, and consistency, and a storage engine is best understood as one particular bundle of answers to those three questions. (from ref-ddia )
2. Ask the shaping question — the book's stated payoff is understanding why data tools are shaped as they are, so when a warehouse refuses a use case, ask which mechanism answer it encodes rather than blaming the vendor. (from ref-ddia )
3. Make consistency concrete — the description maps consistency to transaction semantics, which in practice means asking what your MERGE does under concurrent writers before the engine is chosen, not after the first lost update. (from ref-ddia )
4. Recognize shape patterns in the catalogue — Microsoft's design patterns include CQRS and event sourcing, each a read/write shape decision, so you can name what a serving-table split or a rebuildable log is buying you. (from ref-cloud-design-patterns )
5. Check the choice against deployed guidance — AWS's architecture collection (Well-Architected pillars, data lake patterns) shows which storage bundles ship together, turning your tradeoff argument into precedent. (from ref-aws-arch-center )

Check yourself

• For any table you own, you can state the partitioning scheme, the replication story, and the consistency semantics your MERGE relies on.
• Asked why a lakehouse over a row store, your answer names write shape, scan shape, and time-travel semantics — not brand names.

Worked example

For table T, choose the engine by stating:
- Replication need   -> HA and read scale required? (replication)
- Partitioning need  -> what is the natural partition key? (partitioning)
- Consistency need   -> what must concurrent MERGE writers see? (consistency)
Then name the shape pattern: plain | CQRS-style serving table | event-sourced replay.
Finally cite one deployed reference architecture with the same bundle.

Sourced from

• Designing Data-Intensive Applications (O'Reilly)
• Azure Cloud Design Patterns
• AWS Architecture Center

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Consistency semantics chosen late are chosen wrong — concurrent-writer behavior belongs in the storage decision.
• Every engine bundles all three mechanisms — you never get one answer without paying for the other two.
• CQRS and event sourcing solve read/write shape, not scale — do not reach for them as capacity fixes.

Production considerations

• Write the consistency expectation into the table's README — merges depend on it silently.
• Revisit the storage bundle when the dominant access pattern changes shape.

Diagram

storage engine decision
   |
   |- write shape  --> append? upsert? log?
   |- read shape   --> scans? point lookups? serving tables?
   |- consistency  --> what must a concurrent MERGE see?
   |
   |- mechanisms: replication | partitioning | consistency
   |- shape patterns (CQRS, event sourcing) name the read/write split
   |
    --> choice grounded in a reference architecture that ships the same bundle

Registry stats