DE//AGENT
Explore
Runbook community advanced severity: high

Backfill required

Historical output is wrong or missing and must be recomputed safely, without breaking production.

At a glance

When A past date range is wrong or missing; A logic fix or new column must be applied historically
Where Airflow / Orchestration
Why A backfill is production work with a bigger blast radius than the original pipeline — shadow-write, sample-validate, then swap.

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Symptoms

  1. A past date range is wrong or missing
  2. A logic fix requires historical recompute
  3. A new column needs populating historically

Incident notebook

Impact

The incident is contention risk: an unmanaged backfill can starve daily production runs and double-write history.

Escalation

Schedule with the platform owner; escalate if the backfill window collides with a golden-hour production run.

Evidence to collect first

  • The written definition of correctness and its validation query
  • The lineage-derived downstream impact set
  • The agreed idle window and resource budget

Rollback

Shadow-then-swap makes rollback trivial — delete the shadow, never the production table; swap only after sample validation passes.

Follow-up

  • Batch the dates; validate a sample after each batch
  • Document the runbook so the next backfill is routine

Diagnosis

  • Define correctness: what should the values be, and how will you check?
  • Compute the downstream impact set from lineage.
  • Establish the idle window and the resource budget.

Possible causes

  • A bug that only affected a date range
  • A late upstream correction
  • A new feature that must be applied retroactively

Commands

airflow dags backfill --start-date YYYY-MM-DD --end-date YYYY-MM-DD --reset-dagruns dag_id
-- prefer batched date ranges over one large range

Fix

  • Write to a shadow table, validate a sample, then swap.
  • Batch by date; validate a sample after each batch.
  • Never run a backfill concurrently with the daily production run.

Prevention

  • Keep pipelines idempotent so a backfill is a routine operation.
  • A documented backfill runbook per pipeline.

Monitoring

  • backfill progress, resource contention, validation pass rate

Common gotchas

  • Never run a backfill concurrently with the daily production run.
  • One huge date range starves the cluster; batch it and validate a sample per batch.
  • Backfilling without a written definition of correctness just relocates the bug.

Production considerations

  • Idempotent pipelines make a backfill a routine operation.
  • Document a backfill runbook per pipeline, with its idle window and resource budget.

Tradeoffs

Choose this when Costs & caveats
shadow-then-swap cannot corrupt production double compute until the swap

Diagram

backfill
   |
   |- define correctness --> shadow table --> batch dates --> validate sample --> swap
   |
   |- contention risk --> pick idle window --> cap concurrency

Registry stats

Verification community · Sep 25, 2026
Completeness 75
Trust multiplier ×0.5
Composite 30.25 · adequate
Freshness 29 days since last source check
Difficulty advanced
Severity high
Version 1.1.0
License CC-BY-4.0
Security flags 3