Operate

Production runbooks

Symptom → diagnosis → recovery → prevention for the most common incidents. Triage by severity; every runbook carries commands, prevention and monitoring.

15 runbooks 3 critical9 high3 medium Triage with the incident assistant

Severity

Runbook 29 adequate

Pipeline failed

A task or DAG run has failed and downstream work is blocked — triage, classify and recover.

beginner incidentorchestration community

Runbook 32 adequate

Bad deployment

A deploy broke production; correlate the signal with the change and roll back first.

intermediate deploymentrollback community

Runbook 33 adequate

Data contract broken

A producer violated its contract and a consumer broke in production — bound the impact before fixing.

intermediate contractsla community

Runbook 35 adequate

Data quality failure

A quality expectation failed in production and bad data may be reaching consumers.

intermediate data-qualityquarantine community

Runbook 35 adequate

Duplicate records

Row counts are inflated and metrics double-counted after a retry, a fan-out join or a backfill.

intermediate duplicatesidempotency community

Runbook 29 adequate

Late files

Partner files arrive after the job has already run, so historical numbers keep changing.

intermediate late-dataingestion community

Runbook 29 adequate

Missing partition

An expected date partition is absent, so a downstream report silently excludes a day.

intermediate partitionbatch community

Runbook 35 adequate

Schema drift

The upstream schema changed without notice and a cast broke or a column silently nulled.

intermediate schema-driftcontracts community

Runbook 29 adequate

Small files explosion

A table has fragmented into thousands of tiny files and query planning now dominates runtime.

intermediate small-fileslakehouse community

Runbook 29 adequate

Warehouse query is slow

A query that used to be fast became slow — separate scan inefficiency from compute contention.

intermediate warehousequery community

Runbook 30 adequate

Backfill required

Historical output is wrong or missing and must be recomputed safely, without breaking production.

advanced backfillorchestration community

Runbook 29 adequate

CDC stopped

Change events stopped flowing from a source and the target table is silently diverging.

advanced cdcdebezium community

Runbook 30 adequate

Kafka consumer lagging

A consumer group is falling behind the topic and downstream tables are going stale.

advanced kafkalag community

Runbook 37 adequate

Spark job is slow

A batch Spark job's runtime has jumped — find whether it is skew, partition count, spill or over-scan.

advanced sparkperformance community

Runbook 37 adequate

Spark job suddenly costs 4x

One job's compute cost quadrupled without an obvious functional change — attribute it and reverse the regression.

advanced sparkcost community