DR and replication — design the recovery path before you need it

At a glance

Learning objectives

• Map replication and consistency from DDIA to mechanisms in your own stack
• Keep the old data until the new data is validated — shadow-then-swap as rollback
• Classify recovery per failure class — code rollback vs quarantine-then-retry vs recompute

What you need

• The tables your consumers cannot lose
• The deploy/revert mechanism for pipeline code
• A recovery drill scheduled with the platform owner

Steps

1. DDIA's chapters map directly to patterns in this registry — replication to CDC, partitioning to lakehouse, consistency to transaction semantics. (from ref-ddia )
2. A recovery design must satisfy three forces — correctness must be verifiable, production must stay available, rollback must be possible. (from pattern-backfill-safety )
3. Keep the old data until validated — shadow-then-swap keeps rollback in reach: delete the shadow, never the production table; swap only after sample validation passes. (from runbook-backfill-required )
4. Classify the recovery by failure class — roll back the code deploy if the class is deterministic and new; data errors get quarantine-then-retry, not rollback. (from runbook-pipeline-failed )
5. Recovery is when destructive SQL is most dangerous — destructive statements are denied by default, and a DELETE/UPDATE without a WHERE clause is always denied. (from hook-destructive-sql-guard )
6. Any write-class tool against a production target is suspended and surfaced for a human "approve once" token scoped to the single call. (from hook-prod-write-confirm )

Check yourself

• Every critical table has a named recovery path — code rollback, quarantine-then-retry, or shadow-then-swap recompute.
• The old data still exists until the replacement is validated — you have never deleted production to fix production.

Worked example

Roll back the code deploy if the class is deterministic and new; data errors
get quarantine-then-retry, not rollback.

- Correctness must be verifiable
- Production must stay available
- Rollback must be possible

Sourced from

• Designing Data-Intensive Applications (O'Reilly)
• Safe Backfill
• Backfill required
• Pipeline failed
• Destructive SQL Guard
• Production Write Confirmation

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Roll back the code deploy if the class is deterministic and new — data errors get quarantine-then-retry, not rollback.
• Delete the shadow, never the production table; swap only after sample validation passes.
• A DELETE/UPDATE without a WHERE clause is always denied — recovery scripts are not exempt.

Production considerations

• Map replication to CDC and consistency to transaction semantics when choosing the mechanism for each critical table.
• Suspend write-class tools against production targets during recovery — a human approve-once token scoped to a single call.

Diagram

recovery paths
   |
   |- deterministic (new code) --> roll back the deploy
   |
   |- data error --> quarantine input --> re-run idempotently
   |
   |- recompute --> shadow-then-swap --> old data kept until validated
   |
   |- destructive SQL during recovery --> denied by default

Registry stats