Series · roadmap stage 9/13
The Data Quality Series
Ten lessons in order, from the mental model to a capstone checklist. Every lesson is structured by us and grounded in a real, cited source from the registry — no invented commands.
10 of 10 lessons published
- Lesson 1 Name the four dimensions of data quality
Build the mental model before writing a single test: freshness, completeness, validity, consistency. See each dimension in a real registry artifact — the SLA part of a contract, the profiling workflow of the data-quality reviewer, the rule
- Lesson 2 Deduplicate records without rewriting the table
Row counts are inflated and revenue double-counted after a retry or a fan-out join. Work the duplicate-records runbook: find the first duplicate date, correlate it with a deploy, dedupe on the business key, and rewrite only the affected par
- Lesson 3 Author your first data contract
Draft an enforceable contract for one real dataset by walking the six parts — owner, schema, semantics, SLA, breaking-change policy, enforcement — starting from the data-contract template, and finish with the CI test that makes it a contrac
- Lesson 4 Enforce contracts in CI with the schema change check
Install the Schema Change Check as a pre-commit gate over staged schema files so a breaking change exits non-zero unless a migration plan or deprecation annotation rides along with it — the producer-CI placement that catches breakage before
- Lesson 5 Split a batch into quarantine and publish
Build the pipeline-dq-quarantine starter the way the Quarantine then Publish pattern prescribes: a quality contract splits each batch into a published set and a quarantine with reasons. You finish with a test that proves no row is ever lost
- Lesson 6 Review a schema change against its contract
Run the Schema Reviewer's four-step pass over a producer's schema diff — classify each field change, resolve consumers from lineage, emit a dated migration plan — and use the schema-drift runbook to separate hard failures from silent nulls
- Lesson 7 Gate promotion with a pre-deploy DQ check
Wire the pre-deployment DQ check so a pipeline is only promoted when its quality suite passes against staged output. You attach the pre-deploy trigger, run dbt/GE tests on the staging namespace, and prove a failed expectation leaves product
- Lesson 8 Respond when a data contract breaks
Work the broken-contract runbook end to end: identify which clause broke, bound the consumer impact set from lineage before anything else, decide whether the producer reverts or the consumer adapts, and close with consumer-driven contract t
- Lesson 9 Respond to a failing quality expectation
A quality expectation failed in production and bad data may be reaching consumers. Follow the data-quality-failure runbook: read the failure precisely, quantify it, then make the one decision that matters — block, quarantine, or pass-with-w
- Lesson 10 Capstone — wire a dataset's quality gates end to end
Close the series by walking one SLA-bearing dataset through every gate the series taught — six-part contract, CI compatibility check, staged quality suite, quarantine split, quarantine-rate alerts — and finish with the monitoring signals an