Reliability

The data reliability mental model — freshness, completeness, correctness

Build the map before writing a single check: a consumer experiences a table as three properties — freshness, completeness, correctness — declared as an SLA in a data contract, enforced by a CI test, and spent like an error budget when failures land.

Capstone — the reliability checklist for a pipeline you can page on

Close the series by running the reliability checklist over your own pipeline: a declared and enforced SLA, idempotent tasks with backoff retries, a documented backfill runbook — then drill the incident path from alert to blameless postmortem with dated, owned action items.

DR and replication — design the recovery path before you need it

Disaster recovery for data systems starts before the outage: DDIA's replication and consistency chapters name the mechanisms, and the registry's runbooks name the recovery paths — code rollback for deterministic failures, quarantine-then-retry for data errors, shadow-then-swap for recomputes.

Idempotency and reruns — make every task safe to run twice

Retries are inevitable, so a task that appends duplicates rows every time it is cleared: hold each task to the idempotency test, replace its partition atomically instead of appending, and assert the invariant in CI with a test that runs the same date twice.

Incident Postmortem Writer

• Write a blameless timeline
• Separate trigger from contributing factors

Incident Assistant

Guide an on-call engineer through a data incident and produce the postmortem.