DE//AGENT
Explore
Tutorial Lesson 9 of 10 · The Orchestration Series community intermediate

Plan and execute a safe backfill

Use the Backfill Planner's four-step plan — correctness criteria, downstream impact from lineage, cost and contention, staged batches — then execute it the way the backfill runbook demands: shadow table, sample validation per batch, and never beside the daily production run.

At a glance

When A past date range is wrong or missing; A logic fix or new column must be applied historically
Where Airflow / Orchestration
Why A backfill is production work with a bigger blast radius than the original pipeline — shadow-write, sample-validate, then swap.

Learning objectives

  • Write correctness criteria for a backfill before running it
  • Compute the downstream impact set from lineage
  • Execute the backfill in batches with validation gates and a rollback path

What you need

  • Target table and date range
  • Pipeline dependencies and lineage
  • An idle window that excludes the daily job
  • [object Object]

Steps

  1. Define correctness criteria for the backfill. (from agent-backfill-planner )
  2. Compute downstream impact from lineage. (from agent-backfill-planner )
  3. Estimate cost and contention. (from agent-backfill-planner )
  4. Stage batches with validation between them. (from agent-backfill-planner )
  5. Write to a shadow table, validate a sample, then swap. (from runbook-backfill-required )
  6. Never run a backfill concurrently with the daily production run. (from runbook-backfill-required )

Check yourself

  • The plan includes a rollback path and an explicit window with the daily job excluded — the planner refuses to plan without both.
  • A 90-day backfill comes out planned around the daily run, batched weekly, with a shadow table and a row-count gate.

Worked example

Batched backfill command — from runbook-backfill-required 

airflow dags backfill --start-date YYYY-MM-DD --end-date YYYY-MM-DD --reset-dagruns dag_id
# prefer batched date ranges over one large range

Shadow-write, validate, then swap — from runbook-backfill-required 

- Write to a shadow table, validate a sample, then swap.
- Batch by date; validate a sample after each batch.
- Never run a backfill concurrently with the daily production run.

Sourced from

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

  • Never run a backfill concurrently with the daily production run.
  • One huge date range starves the cluster; batch it and validate a sample per batch.
  • Backfilling without a written definition of correctness just relocates the bug.

Production considerations

  • Idempotent pipelines make a backfill a routine operation.
  • Document a backfill runbook per pipeline, with its idle window and resource budget.

Diagram

backfill plan
   |
   |- correctness criteria --> downstream impact from lineage
   |
   |- cost + contention --> idle window with daily job excluded
   |
   |- shadow table --> batched dates --> validate sample per batch --> swap

Registry stats

Verification community · Oct 21, 2026
Completeness 81
Trust multiplier ×1.25
Composite 34.15 · adequate
Freshness 3 days since last source check
Difficulty intermediate
Version 0.1.0
License CC-BY-4.0
Security flags none