Production gotchas — silent failures and partial writes

At a glance

Learning objectives

• Catch silent failures — hidden cost regressions and silently widened freshness SLAs
• Stop partial writes — no appends to recomputable partitions, no unguarded prod writes
• Put the failure class in the alert text so on-call can triage before touching anything

What you need

• The pipelines currently in production
• An alerting channel wired to task failure rate and SLA misses
• CI where a hook can warn or block a merge

Steps

1. A working pipeline silently becomes expensive and nobody notices until the invoice. (from pattern-cost-guardrails )
2. A change can silently widen the freshness SLA of a source — compare the schedule/cron in the diff with the source's declared freshness SLA and warn with the computed worst-case lag. (from hook-freshness-pre-merge )
3. Partial writes come from appends — retries and backfills duplicate or corrupt data because writes are appends; replace the partition atomically instead. (from pattern-idempotent-partition-overwrite )
4. Read the first real error in the log, not the last — and retrying a data error re-runs the same failure; fix or quarantine the input first. (from runbook-pipeline-failed )
5. Guard every production write — any write-class tool against a production target is suspended and surfaced for a human "approve once" token scoped to the single call. (from hook-prod-write-confirm )
6. `DROP TABLE prod.orders` from an agent is denied and surfaced for confirmation. (from hook-destructive-sql-guard )
7. Retry with backoff, and put the failure class in the alert text. (from runbook-pipeline-failed )

Check yourself

• A cron change that would violate the freshness SLA warns at pre-merge with the computed worst-case lag.
• No production write path exists that a hook does not classify, confirm or block.

Worked example

Changing a source cron from hourly to daily warns that the 6h freshness SLA
will be violated.

Classify the tool target as production by connection string, project or
catalogue tag. Any write-class tool against a production target is suspended
and surfaced for a human "approve once" token scoped to the single call.

Sourced from

• Cost Guardrails
• Freshness Pre-Merge Check
• Idempotent Partition Overwrite
• Pipeline failed
• Production Write Confirmation
• Destructive SQL Guard

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Read the first real error in the log, not the last.
• Retrying a data error re-runs the same failure; fix or quarantine the input first.
• Spend varies with data growth and regressions hide in growth — total-spend dashboards cannot see them.

Production considerations

• Retry with backoff, and put the failure class in the alert text.
• Make every task idempotent so a re-run is always safe.

Diagram

silent failures + partial writes
   |
   |- regression hides in growth --> unit metric + baseline catches it
   |
   |- cron change --> freshness SLA widened silently --> warn with worst-case lag
   |
   |- append on retry --> duplicated rows --> atomic partition replace
   |
   |- prod write --> classified + confirmed or blocked

Registry stats