DE//AGENT
Explore
Tutorial Lesson 8 of 10 · The Orchestration Series community intermediate

Production gotchas — topology, timeouts, concurrency

The DAG failures that become pages: unbounded concurrency, missing catchup policy, and a backfill that starves the daily run — cap them with max_active_runs and pools, then wire the monitoring and escalation path the backfill runbook prescribes.

At a glance

When Hardening a DAG before it takes production traffic; Planning a backfill that runs beside live schedules
Where Airflow / Orchestration
Why Unbounded concurrency and a missing catchup policy only surface when a backfill collides with the daily run — cap both first.

Learning objectives

  • Cap max_active_runs and pool limits before a backfill starves the daily run
  • Find unsafe retries, missing catchup policy and unbounded concurrency
  • Monitor backfill progress, resource contention and validation pass rate

What you need

  • The DAGs currently in production
  • The daily production schedule and golden-hour windows
  • [object Object]

Steps

  1. Check the DAG for unsafe retries, missing catchup policy, ambiguous catchup/backfill semantics and unbounded concurrency. (from skill-airflow-dag-reviewer )
  2. Cap max_active_runs and pool limits so a backfill cannot starve the daily run. (from skill-airflow-dag-reviewer )
  3. Never overlap the production window; never backfill without a rollback path. (from pattern-backfill-safety )
  4. Design against the incident — an unmanaged backfill can starve daily production runs and double-write history. (from runbook-backfill-required )
  5. Monitor backfill progress, resource contention and validation pass rate. (from runbook-backfill-required )
  6. Schedule with the platform owner; escalate if the backfill window collides with a golden-hour production run. (from runbook-backfill-required )

Check yourself

  • max_active_runs and pool limits are set on every DAG a backfill could touch.
  • The escalation path names the platform owner and the golden-hour window that must stay clear.

Worked example

Concurrency guard from the checklist — from skill-airflow-dag-reviewer 

`max_active_runs` and pool limits prevent a backfill from starving the daily run.

What the reviewer flags — from skill-airflow-dag-reviewer 

Non-idempotent tasks, unsafe retries, missing catchup policy, ambiguous
catchup/backfill semantics and unbounded concurrency.

Sourced from

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

  • Cap max_active_runs and pool limits so a backfill cannot starve the daily run.
  • Never overlap the production window; never backfill without a rollback path.
  • Ambiguous catchup/backfill semantics are a review finding, not a footnote.

Production considerations

  • Monitor backfill progress, resource contention and validation pass rate.
  • Escalate if the backfill window collides with a golden-hour production run.

Diagram

production topology
   |
   |- max_active_runs + pools --> contention capped
   |
   |- catchup policy --> catchup/backfill semantics explicit
   |
   |- monitoring --> progress, contention, validation pass rate

Registry stats

Verification community · Oct 23, 2026
Completeness 81
Trust multiplier ×1.25
Composite 34.15 · adequate
Freshness 1 days since last source check
Difficulty intermediate
Version 0.1.0
License CC-BY-4.0
Security flags none