DE//AGENT
Explore
Runbook community intermediate severity: critical

Bad deployment

A deploy broke production; correlate the signal with the change and roll back first.

At a glance

When Errors or data changes began right after a deploy; A quality check degraded right after promotion
Where CI/CD
Why Roll back first, diagnose at leisure — every extra hour of a bad model in production is wrong data shipped to consumers.

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Symptoms

  1. Errors or data changes began immediately after a deploy
  2. A metric or quality check degraded right after promotion
  3. A previous-good version is known

Incident notebook

Impact

Errors or wrong data from the moment of promotion; the blast radius grows with every scheduled run that picks up the bad artifact.

Escalation

Page the on-call and the deploying team; declare an incident when quality gates degrade post-promotion.

Evidence to collect first

  • The deploy timestamp vs the first failing signal
  • git log / dbt state:modified+ diff identifying the changed artifact
  • Confirmation the previous version is a valid rollback target

Rollback

Roll back the artifact FIRST, diagnose after; if bad data was written, also re-run the affected period from the rolled-back code.

Follow-up

  • Move tests to a staged build in CI, not production
  • Add deploy markers and post-deploy quality gates to the dashboard

Diagnosis

  • Correlate the first failing signal with the deployment timestamp.
  • Identify the exact changed artifact and its blast radius.
  • Confirm the previous version is a valid rollback target.

Possible causes

  • A logic regression shipped without a test catching it
  • A config/env change (e.g. timezone, cluster setting)
  • An incompatible schema or model change

Commands

git log --since='2 hours ago' --oneline
dbt ls --select state:modified+  # what changed in the last build

Fix

  • Roll back the artifact first, then diagnose at leisure.
  • If data was written, roll back the code and re-run the affected period.

Prevention

  • [object Object]
  • [object Object]

Monitoring

  • deploy markers on dashboards, post-deploy quality gates

Common gotchas

  • Roll back before you root-cause; a known-good target is worth more than the diagnosis.
  • Config changes count: timezone or cluster settings produce invisible regressions.
  • If bad data was written, rolling back code isn't enough — re-run the affected period.

Production considerations

  • Run tests against a staged build in CI, not production.
  • Deploy pipelines like services: small diffs, reversible, observable.

Tradeoffs

Choose this when Costs & caveats
rollback restores consumers in minutes rollback discards any good changes shipped in the same window

Diagram

bad deployment
   |
   |- first failing signal --> deploy timestamp --> exact changed artifact
   |
   |- previous-good exists --> rollback --> diagnose --> re-run affected period

Registry stats

Verification community · Sep 20, 2026
Completeness 75
Trust multiplier ×0.75
Composite 32.25 · adequate
Freshness 34 days since last source check
Difficulty intermediate
Severity critical
Version 1.0.1
License CC-BY-4.0
Security flags 2