DE//AGENT
Explore
Project community intermediate

Pipeline → DQ → quarantine → publish

Put a contract in front of a table: validate each batch, publish the clean rows, quarantine the rest with reasons, and alert on the quarantine rate.

At a glance

Where Data Quality · tech-spark · tech-duckdb · tech-parquet

Learning objectives

  • Express a quality contract as executable checks
  • Design a quarantine table with a reason
  • Alert on the quarantine rate, not just on failure

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Implementation

  • Load the contract, compile rules to SQL (DuckDB) or Spark expressions.
  • Split into pass/fail with a stable `failure_reason`.
  • Publish the pass set; write the fail set to quarantine with provenance.

Tests

  • every row is either published or quarantined — never lost
  • the quarantine contains exactly the seeded bad rows
  • a clean batch produces an empty quarantine

Failure Scenarios

  • Contract too strict → quarantine rate explodes; tier rules and loosen with evidence
  • Quarantine never reviewed → add a scheduled review and a reprocess path

Production Hardening

  • Alert when quarantine rate > 1% for 2 consecutive runs
  • A reprocess command that re-runs the quarantined rows after a fix

Registry stats

Verification community · Sep 26, 2026
Completeness 81
Trust multiplier ×0.75
Composite 38.65 · adequate
Freshness 28 days since last source check
Difficulty intermediate
Version 1.0.0
License MIT
Security flags 2

Repository Structure

pipeline-dq-quarantine/
  contract/orders.yml       # rules: not_null, unique, range, enum
  src/validate.py
  src/publish.py
  tests/test_quarantine.py

Architecture

batch ─▶ [contract check] ─┬─ pass ─▶ published/ (SLA-bearing) └─ fail ─▶ quarantine/ (+reason, +source_file) │ └─▶ alert on rate

Setup

  • make test # validates against the fixtures offline
  • make run

Sample Data

A batch with ~5% deliberately invalid rows (nulls, dupes, out-of-range).