DE//AGENT
Explore
Tutorial Lesson 3 of 10 · The Orchestration Series community intermediate

Wire task dependencies and retries that survive failure

Model the pipeline as task groups and give every task deliberate retries with backoff, a timeout and a pool — the skeleton's guarantees, verified the way the DAG reviewer verifies them before the DAG ships.

At a glance

When Wiring task dependencies and retry settings in a new or inherited DAG; A task retries instantly and hammers a downstream system
Where Airflow / Orchestration
Why Dependencies decide what runs with what, retries decide what happens when it fails — take both from the skeleton instead of rediscovering them per task.

Learning objectives

  • Model task structure with task groups from the idempotent skeleton
  • Choose retries and timeouts deliberately, with backoff
  • Pass the reviewer's retry and concurrency check

What you need

  • A scaffolded DAG or any DAG file under review
  • The pipeline's SLA
  • [object Object]

Steps

  1. Scaffold from the idempotent DAG skeleton — task groups, retries and backoff, SLA, pools, and a sensor using reschedule mode are built in. (from tpl-airflow-dag )
  2. Choose retries and timeouts deliberately. (from skill-airflow-dag-reviewer )
  3. Retries have a backoff; a task that retries instantly will hammer a downstream. (from skill-airflow-dag-reviewer )
  4. Check retry, timeout and concurrency settings for every task. (from agent-pipeline-reviewer )
  5. Run the reviewer over the DAG — it flags unsafe retries, missing catchup policy and unbounded concurrency. (from skill-airflow-dag-reviewer )
  6. Make every task safe to re-run before adding the next one. (from skill-airflow-dag-reviewer )

Check yourself

  • Every task in the DAG declares retries with backoff, a timeout and a pool.
  • Running the reviewer over the DAG returns no unsafe-retry or unbounded-concurrency findings.

Worked example

What the reviewer checks — from skill-airflow-dag-reviewer 

Checks a DAG for non-idempotent tasks, unsafe retries, missing catchup
policy, ambiguous catchup/backfill semantics and unbounded concurrency.

Scaffold with the guarantees — from tpl-airflow-dag 

amp template new de-agent/airflow-dag ./dags

Sourced from

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

  • Retries have a backoff; a task that retries instantly will hammer a downstream.
  • The reviewer checks for non-idempotent tasks, unsafe retries, missing catchup policy and unbounded concurrency.

Production considerations

  • SLA, pools and backoff come from the template — keep them instead of stripping defaults.
  • Make every task safe to re-run before adding the next one.

Diagram

dependencies + retries
   |
   |- task groups --> pipeline structure
   |
   |- retries + backoff --> failure handling
   |
   |- timeouts + pools + SLA --> bounded resource use

Registry stats

Verification community · Oct 23, 2026
Completeness 81
Trust multiplier ×1.25
Composite 34.15 · adequate
Freshness 1 days since last source check
Difficulty intermediate
Version 0.1.0
License CC-BY-4.0
Security flags none