Idempotency and reruns — make every task safe to run twice

At a glance

Learning objectives

• Apply the idempotency test to every task before trusting retries
• Replace appends with atomic partition overwrites keyed by the logical date, or MERGE
• Assert idempotency in CI with an invariant test on row counts

What you need

• A batch job that can be cleared and retried
• A logical date or partition key per run
• [object Object]

Steps

1. Retries and backfills duplicate or corrupt data because writes are appends. (from pattern-idempotent-partition-overwrite )
2. Hold every task to the idempotency test — "If this task runs twice for the same logical date, is the output the same?" If no, fix it before anything else — retries will happen. (from skill-airflow-dag-reviewer )
3. Make each run write the complete result for its logical partition and replace that partition atomically — `INSERT OVERWRITE partition(dt=...)` or Delta/Iceberg replace-where. The invariant — running the same date twice yields identical output. (from pattern-idempotent-partition-overwrite )
4. Every task writes to a partition keyed by the logical date, or uses MERGE. (from skill-airflow-dag-reviewer )
5. A re-run is only safe if the task is idempotent — check before clearing. (from runbook-pipeline-failed )
6. Make every task idempotent so a re-run is always safe, and retry with backoff. (from runbook-pipeline-failed )
7. Assert invariants, not snapshots — count(*) is stable across a re-run (idempotency), no nulls in the grain key, sum(revenue) equals the source sum within tolerance. (from skill-pipeline-test-author )

Check yourself

• Running the same logical date twice yields identical output — proven by a CI test that asserts count(*) is stable across a re-run.
• No task appends to a partition it can recompute.

Worked example

"If this task runs twice for the same logical date, is the output the same?"
If no, fix it before anything else — retries will happen.

- `count(*)` is stable across a re-run (idempotency)
- no nulls in the grain key
- sum(revenue) equals the source sum within tolerance
- a quarantined row never appears in the published table

Make each run write the complete result for its logical partition and replace
that partition atomically. Never append to a partition you can recompute.
`INSERT OVERWRITE partition(dt=...)` or Delta/Iceberg replace-where do this.

Sourced from

• Idempotent Partition Overwrite
• Airflow DAG Reviewer
• Pipeline failed
• Pipeline Test Author

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Retries and backfills duplicate or corrupt data because writes are appends.
• A re-run is only safe if the task is idempotent — check before clearing.
• Idempotency costs an upsert/overwrite pattern on every write — pay it once and retries become routine.

Production considerations

• Idempotent tasks turn incidents into routine re-runs.
• If a test needs the network or a running warehouse, it is not a unit test — keep the invariant harness offline.

Diagram

idempotent reruns
   |
   |- idempotency test --> "same logical date twice, same output?"
   |
   |- append --> duplicates on retry   (the bug)
   |- INSERT OVERWRITE partition / MERGE --> atomic replace  (the fix)
   |
   |- CI invariant --> count(*) stable across re-run

Registry stats