Split a batch into quarantine and publish

At a glance

Learning objectives

• Express a quality contract as executable checks
• Design a quarantine table with a reason
• Alert on the quarantine rate, not just on failure

What you need

• The pipeline-dq-quarantine starter (contract/orders.yml, src/validate.py, src/publish.py)
• DuckDB or PySpark to compile the contract rules to SQL
• A batch with ~5% deliberately invalid rows (nulls, dupes, out-of-range)
• [object Object]

Steps

1. Split every batch into publish and quarantine on a quality contract. (from pattern-quarantine-publish )
2. Validate the contract offline against fixtures with `make test`, then run `make run`. (from project-pipeline-dq-quarantine )
3. Load the contract and compile rules to SQL (DuckDB) or Spark expressions. (from project-pipeline-dq-quarantine )
4. Split into pass/fail with a stable `failure_reason`. (from project-pipeline-dq-quarantine )
5. Publish the pass set; write the fail set to quarantine with provenance. (from project-pipeline-dq-quarantine )
6. Alert when the quarantine rate > 1% for 2 consecutive runs. (from project-pipeline-dq-quarantine )

Check yourself

• Every row is either published or quarantined — never lost; the quarantine contains exactly the seeded bad rows, and a clean batch produces an empty quarantine.
• A quarantine rate above 1% for 2 consecutive runs fires the alert with the failure_reason breakdown attached.

Worked example

Split every batch into publish and quarantine on a quality contract.
  pass ─▶ published table (SLA-bearing)
  fail ─▶ quarantine table (retained, alertable, reprocessable)
A row never disappears: it is either published or quarantined with a reason.
Alert on quarantine *rate*, not just on a hard failure — a rising rate is the
early signal of an upstream problem.

pipeline-dq-quarantine/
  contract/orders.yml       # rules: not_null, unique, range, enum
  src/validate.py
  src/publish.py
  tests/test_quarantine.py

make test  # validates against the fixtures offline
make run

Sourced from

• Quarantine then Publish
• Pipeline → DQ → quarantine → publish

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• A quarantine nobody reviews is just a slower data loss.
• Contract too strict → quarantine rate explodes; tier rules and loosen with evidence.
• Alert on quarantine rate, not just on a hard failure — a rising rate is the early signal of an upstream problem.

Production considerations

• Alert when quarantine rate > 1% for 2 consecutive runs.
• Ship a reprocess command that re-runs the quarantined rows after a fix.

Diagram

batch
   |
   |- [contract check] --> pass --> published/ (SLA-bearing)
   |
   |- [contract check] --> fail --> quarantine/ (+reason, +source_file)
   |
   |- quarantine rate > 1% x2 runs --> alert --> review + reprocess

Registry stats