Scaffold an idempotent DAG from the template

At a glance

Learning objectives

• Scaffold a DAG project with tests and a Dockerfile from the template
• Write tasks that are safe to re-run from day one
• Set catchup, retries and sensor modes as deliberate choices

What you need

• An Airflow development environment
• [object Object]
• A pipeline to model, such as a daily orders job

Steps

1. Run amp template new de-agent/airflow-dag ./dags. (from tpl-airflow-dag )
2. Work in the generated layout: dags/orders_pipeline.py, plugins/, tests/test_dag_integrity.py, Dockerfile. (from tpl-airflow-dag )
3. Keep the skeleton's guarantees — task groups, retries and backoff, SLA, pools, and a sensor using reschedule mode. (from tpl-airflow-dag )
4. Every task writes to a partition keyed by the logical date, or uses MERGE. (from skill-airflow-dag-reviewer )
5. Set `catchup` as a deliberate choice, not a default. (from skill-airflow-dag-reviewer )

Check yourself

• The generated project matches the template layout — dags/orders_pipeline.py, plugins/, tests/test_dag_integrity.py and a Dockerfile are all present.
• Asking the idempotency question of each task — same logical date, same output? — every task answers yes.

Worked example

amp template new de-agent/airflow-dag ./dags

airflow-dag/
  dags/orders_pipeline.py
  plugins/
  tests/test_dag_integrity.py
  Dockerfile

1. Every task writes to a partition keyed by the logical date, or uses MERGE.
2. `catchup` is a deliberate choice, not a default.
3. Retries have a backoff; a task that retries instantly will hammer a downstream.
4. `max_active_runs` and pool limits prevent a backfill from starving the daily run.
5. Sensors use `reschedule`, not `poke`, so they don't hold a worker slot.

Sourced from

• Airflow DAG Template
• Airflow DAG Reviewer
• Safe Backfill

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• `max_active_runs` and pool limits prevent a backfill from starving the daily run.
• Sensors use `reschedule`, not `poke`, so they don't hold a worker slot.
• Never overlap the production window; never backfill without a rollback path.

Production considerations

• The template ships tests/test_dag_integrity.py — run DAG-integrity tests in CI.
• Pair the scaffold with skill-airflow-dag-reviewer and pattern-backfill-safety, the template's declared related resources.

Diagram

scaffold dag
   |
   |- amp template new --> orders_pipeline.py + tests + Dockerfile
   |
   |- keep guarantees --> retries/backoff, SLA, pools, reschedule sensor
   |
   |- partition or MERGE writes --> deliberate catchup --> review with idempotency test

Registry stats