Airflow / Orchestration

Review a DAG for idempotency and retry safety

Walk the Pipeline Reviewer's four-step pass over a DAG: apply the idempotency test to every task, check retry, timeout and concurrency settings, predict backfill behaviour, and propose missing integration tests — then close the loop with the Airflow DAG Reviewer's checklist so every task is safe to re-run before the DAG merges.

Scaffold an idempotent DAG from the template

Generate a new pipeline from the Airflow DAG template and keep its guarantees — an idempotent skeleton with task groups, retries and backoff, SLA, pools, and a reschedule-mode sensor — then make catchup a deliberate choice and every write partition-keyed or a MERGE.

The orchestration mental model — DAGs, schedules, idempotency, backfills

Build the map before touching code: a DAG plus its schedule decides when runs happen, the idempotency test decides whether re-runs are safe, catchup decides what happens to missed intervals, and a backfill decides how history gets fixed — four decisions every later lesson sharpens.

Capstone — the checklist that decides whether a DAG ships

Close the series by running the full review pass over your own pipeline: the idempotency test per task, partitioned or MERGE writes, deliberate catchup, backoff retries, reschedule sensors, bounded concurrency — and a documented runbook for the day history needs fixing.

Fan out over date batches with dynamic task mapping

Run one task definition over many date batches with dynamic task mapping — the docs' mapping feature applied the way the backfill material demands: batched ranges, smallest first, a validated sample after each batch.

Gate backfills with the safety check hook

Wire the Backfill Safety Check as a pre-run gate that refuses any backfill lacking an explicit date range, a shadow or snapshotted target, and a window clear of the scheduled production run — and watch it block the bare backfill command that would otherwise corrupt the target.

Make a DAG data-aware with reschedule-mode sensors

Stop guessing when upstream data arrives: add a sensor in reschedule mode so the DAG waits for its input without holding a worker slot, read the scheduling-semantics pages first, and bound the sensor with timeout and concurrency settings like any other task.

Plan and execute a safe backfill

Use the Backfill Planner's four-step plan — correctness criteria, downstream impact from lineage, cost and contention, staged batches — then execute it the way the backfill runbook demands: shadow table, sample validation per batch, and never beside the daily production run.

Production gotchas — topology, timeouts, concurrency

The DAG failures that become pages: unbounded concurrency, missing catchup policy, and a backfill that starves the daily run — cap them with max_active_runs and pools, then wire the monitoring and escalation path the backfill runbook prescribes.

Wire task dependencies and retries that survive failure

Model the pipeline as task groups and give every task deliberate retries with backoff, a timeout and a pool — the skeleton's guarantees, verified the way the DAG reviewer verifies them before the DAG ships.

Airflow DAG Reviewer

• Make every task safe to re-run
• Choose retries and timeouts deliberately

Backfill Planner

• Scope a backfill by date and by dependency
• Detect which downstream tables must also be rebuilt

Backfill Safety Check

Refuse to start a backfill that lacks a rollback path or overlaps the daily run.

Pipeline Reviewer

Review a DAG or pipeline for idempotency, retries and backfill safety.

Backfill Planner

Produce a safe, staged backfill plan with validation and rollback.

Airflow DAG Template

An idempotent DAG skeleton with task groups, retries and backoff, SLA, pools, and a sensor using reschedule mode.

Backfill required

Historical output is wrong or missing and must be recomputed safely, without breaking production.

Safe Backfill

Backfills corrupt production or take down the daily run.

Apache Airflow Documentation

Airflow's documentation — DAG authoring, scheduling semantics and timetables, backfills and catchup, task state and retries, dynamic task mapping, and best practices. The scheduling and best-practices pages answer the idempotency and backfill questions our orchestration skills only sketch; read them before designing any recurring pipeline.