DE//AGENT
Explore
Runbook community advanced severity: critical

CDC stopped

Change events stopped flowing from a source and the target table is silently diverging.

At a glance

When Change events stopped arriving from a source table; The target is silently diverging and consumers see no error
Where CDC
Why CDC failures are silent by construction — the target keeps serving yesterday's truth, and divergence compounds until a reconciliation catches it.

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Symptoms

  1. Change events stop arriving for a source table
  2. Target table silently diverges from source
  3. Often no error surfaced to consumers

Incident notebook

Impact

The target silently diverges from the source — every consumer reads stale data with no error anywhere; reports are wrong by omission.

Escalation

Declare an incident immediately; page the platform on-call and the source-database owner (slot and WAL changes need DBA rights).

Evidence to collect first

  • Connector and task status for the affected source
  • pg_replication_slots: active, restart_lsn, confirmed_flush_lsn, retained WAL
  • Topic offsets: last event timestamp vs the source's max LSN/SCN
  • Source-to-target row-count delta for the affected tables

Rollback

Not applicable — recovery is forward-only: restart the task, or re-snapshot if the slot/WAL was lost. Never hand-edit offsets.

Follow-up

  • Add the retained-WAL and task-state alerts that were missing
  • Run a full reconciliation over the divergence window
  • Postmortem: why monitoring missed the silence

Diagnosis

  • Check connector and task status for the source.
  • Check the replication slot's retained WAL / active state on the DB.
  • Confirm the topic still has recent timestamps.

Possible causes

  • Connector task failed (slot invalidated, WAL purged)
  • Replication slot dropped or lagging beyond retention
  • Network/credentials rotated and the connector cannot reconnect
  • Schema change broke the connector's mapping

Commands

curl -s $CONNECT/connectors/$NAME/status | jq .tasks
SELECT slot_name, active, restart_lsn, confirmed_flush_lsn FROM pg_replication_slots;
kafka-run-class kafka.tools.GetOffsetShell --topic $T --time -1

Fix

  • Task failed → restart the task; if the slot was dropped, re-snapshot.
  • WAL purged → the only correct fix is a fresh snapshot + restart.
  • Never hand-edit offsets; replay from the snapshot point.

Prevention

  • Alert on replication-slot retained WAL and on connector task failures.
  • Monitor the source→target row-count delta continuously.

Monitoring

  • connector task state, slot retained WAL, event lag vs source

Common gotchas

  • Never hand-edit offsets; replay from the snapshot point.
  • WAL purged beyond the slot's retention is unrecoverable in place — the only correct fix is a fresh snapshot.
  • Credential rotation can stall a connector with no error on the consumer side.

Production considerations

  • Alert on replication-slot retained WAL and on connector task state.
  • Reconcile source-to-target row counts continuously.

Tradeoffs

Choose this when Costs & caveats
a fresh snapshot is always correct full re-snapshotting is the most expensive recovery path

Diagram

cdc stopped
   |
   |- task failed --> restart --> slot intact? --> resume
   |                                 |- slot dropped --> re-snapshot
   |
   |- WAL purged --> fresh snapshot --> restart from snapshot point

Registry stats

Verification community · Sep 24, 2026
Completeness 75
Trust multiplier ×0.5
Composite 29.25 · adequate
Freshness 30 days since last source check
Difficulty advanced
Severity critical
Version 1.1.0
License CC-BY-4.0
Security flags 3