Plan and execute a backfill that doesn't break production

At a glance

Learning objectives

• Scope a backfill by date, dependency and blast radius before running anything
• Shadow-write in date batches, smallest first, validating a sample per batch
• Gate the backfill on an explicit range and a rollback path

What you need

• A past date range in a production table that is wrong or missing
• The pipeline's idle window and resource budget
• [object Object]

Steps

1. Historical output is wrong or missing and must be recomputed safely, without breaking production. (from runbook-backfill-required )
2. Before running anything — which dates are wrong and what does "correct" look like; every downstream model/table that reads the target; will this run beside the daily job and at what hour the cluster is idle; can each date be re-run safely; snapshot the target or write to a shadow table first. (from skill-backfill-planner )
3. Define correctness first — what should the values be, and how will you check? Then compute the downstream impact set from lineage. (from runbook-backfill-required )
4. Treat a backfill as a controlled migration — 1. shadow target (or snapshot), 2. batched date ranges smallest first, 3. validate a sample after each batch, 4. swap atomically, 5. keep the old data until validated. (from pattern-backfill-safety )
5. The safety gate requires an explicit date range, a target that is either a shadow table or snapshotted, and a window that does not overlap the scheduled production run — it denies otherwise. (from hook-backfill-safety-check )
6. Never run a backfill concurrently with the daily production run; one huge date range starves the cluster — batch it and validate a sample per batch. (from runbook-backfill-required )
7. Shadow-then-swap keeps rollback in reach — delete the shadow, never the production table; swap only after sample validation passes. (from runbook-backfill-required )

Check yourself

• The plan names the wrong dates, the written definition of correct, the downstream blast radius and the rollback path before anything runs.
• The run writes to a shadow target in date batches, smallest first, and the swap happens only after sample validation passes.

Worked example

airflow dags backfill --start-date YYYY-MM-DD --end-date YYYY-MM-DD --reset-dagruns dag_id
# prefer batched date ranges over one large range

Intercepts a backfill trigger. Requires: an explicit date range, a target that
is either a shadow table or snapshotted, and a window that does not overlap the
scheduled production run. Denies otherwise.

1. shadow target (or snapshot)
2. batched date ranges, smallest first
3. validate a sample after each batch
4. swap atomically
5. keep the old data until validated

Sourced from

• Backfill required
• Backfill Planner
• Safe Backfill
• Backfill Safety Check

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Never run a backfill concurrently with the daily production run.
• One huge date range starves the cluster; batch it and validate a sample per batch.
• Backfilling without a written definition of correctness just relocates the bug.

Production considerations

• Idempotent pipelines make a backfill a routine operation.
• Document a backfill runbook per pipeline, with its idle window and resource budget.

Diagram

safe backfill
   |
   |- define correct + lineage impact --> blast radius known
   |
   |- shadow target (or snapshot) --> batched ranges, smallest first
   |
   |- validate sample per batch --> pass --> swap atomically
   |                              fail --> delete the shadow, never prod

Registry stats