Incident — the numbers doubled and no alarm fired

At a glance

Learning objectives

• Recognize inflated row counts and double-counted metrics as the fan-out signature
• Count duplicates per key and correlate the first duplicate date with a deploy
• Rewrite affected partitions instead of the whole table

What you need

• A table whose counts jumped and the pipeline that wrote it
• Write access to rewrite the affected partitions
• [object Object]

Steps

1. Name the incident — row counts are inflated and metrics double-counted after a retry, a fan-out join or a backfill. (from runbook-duplicate-records )
2. Identify the duplicate key and count duplicates per key. (from runbook-duplicate-records )
3. Run: SELECT k, count(*) c FROM t GROUP BY k HAVING count(*) > 1 ORDER BY c DESC LIMIT 20; (from runbook-duplicate-records )
4. Determine the earliest date duplicates appeared → correlate with a deploy. (from runbook-duplicate-records )
5. Check whether the write is append vs merge/overwrite. (from runbook-duplicate-records )
6. Deduplicate on the business key, keeping the latest by event time. (from runbook-duplicate-records )
7. Rewrite the affected partitions (overwrite), not the whole table. (from runbook-duplicate-records )

Check yourself

• The GROUP BY duplicate count returns zero rows after the fix, on the affected partitions.
• You can name the deploy that introduced the duplicates and the date it landed.
• The rewrite touched only the partitions with duplicates — the rest of the table is untouched.

Worked example

SELECT k, count(*) c FROM t GROUP BY k HAVING count(*) > 1 ORDER BY c DESC LIMIT 20;
SELECT min(dt), max(dt) FROM (SELECT dt, k, count(*) FROM t GROUP BY dt, k HAVING count(*)>1);

Sourced from

• Duplicate records
• SQL Review

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• The earliest duplicate date, not the loudest metric, points at the causing deploy.
• Append vs merge/overwrite decides whether the fix is a dedupe or a re-run.
• Rewriting the whole table for one bad day widens the blast radius.

Production considerations

• Alert on duplicates per key, not just on row-count deltas.
• Prevent the recurrence with the review rule: prove a join cannot fan out before writing it.

Diagram

silent fan-out incident
   |
   |- signature --> counts inflated, metrics double-counted
   |
   |- GROUP BY HAVING --> duplicates per key
   |
   |- earliest duplicate date --> correlate with deploy
   |
   |- append vs merge/overwrite --> dedupe on business key
   |
   |- rewrite affected partitions --> verify duplicate count returns to zero

Registry stats