The orchestration mental model — DAGs, schedules, idempotency, backfills

At a glance

Learning objectives

• Name the four decisions orchestration makes — schedule, catchup, idempotency, backfill
• Apply the idempotency test to a task before running anything
• Say what a backfill is for and why it is dangerous

What you need

• An Airflow project to reason about — real or planned
• The pipeline's schedule and SLA
• The Airflow documentation at airflow.apache.org/docs

Steps

1. Airflow's documentation covers DAG authoring, scheduling semantics and timetables, backfills and catchup, task state and retries, dynamic task mapping, and best practices — that list is the map of orchestration. (from ref-airflow-docs )
2. Read the scheduling and best-practices pages before designing any recurring pipeline — they answer the idempotency and backfill questions the skills only sketch. (from ref-airflow-docs )
3. Set `catchup` as a deliberate choice, not a default. (from skill-airflow-dag-reviewer )
4. Hold every task to the idempotency test — "If this task runs twice for the same logical date, is the output the same?" (from skill-airflow-dag-reviewer )
5. If no, fix it before anything else — retries will happen. (from skill-airflow-dag-reviewer )
6. A backfill exists because historical output is wrong or missing and must be recomputed safely, without breaking production. (from runbook-backfill-required )
7. Respect its blast radius — backfills corrupt production or take down the daily run. (from pattern-backfill-safety )

Check yourself

• You can say what a DAG, a logical date, `catchup` and a backfill each decide about a pipeline.
• Asked whether a task is safe to re-run, you reach for the idempotency test instead of a guess.
• Given a wrong historical date range, you name the backfill runbook before touching anything.

Worked example

"If this task runs twice for the same logical date, is the output the same?"
If no, fix it before anything else — retries will happen.

DAG authoring, scheduling semantics and timetables, backfills and catchup,
task state and retries, dynamic task mapping, and best practices.

Sourced from

• Apache Airflow Documentation
• Airflow DAG Reviewer
• Backfill required
• Safe Backfill

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• `catchup` is a deliberate choice, not a default.
• Retries will happen — fix a task that is not idempotent before anything else.
• Backfills corrupt production or take down the daily run when run unmanaged.

Production considerations

• The scheduling and best-practices pages answer the idempotency and backfill questions the skills only sketch.
• Keep pipelines idempotent so a backfill is a routine operation.

Diagram

orchestration mental model
   |
   |- DAG + schedule + catchup --> when runs happen
   |
   |- idempotency test --> whether re-runs are safe
   |
   |- backfill + shadow swap --> how history gets fixed

Registry stats