DE//AGENT
Explore
Runbook community advanced severity: high

Kafka consumer lagging

A consumer group is falling behind the topic and downstream tables are going stale.

At a glance

When Consumer-group lag keeps rising and won't recover on its own; Downstream tables are going fresh-to-stale
Where Kafka
Why Lag compounds — every minute behind is a minute of stale truth downstream, and the sink silently becomes the bottleneck nobody owns.

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Symptoms

  1. Consumer group lag rising and not recovering
  2. Downstream tables increasingly stale
  3. Processor CPU may be low (waiting on downstream) or saturated

Incident notebook

Impact

Downstream tables go stale minute by minute; freshness SLAs breach and consumers act on old data without knowing it.

Escalation

Page the streaming on-call when the lag slope stays positive for 3 consecutive alert intervals; loop in the sink's owner early — the bottleneck is usually downstream.

Evidence to collect first

  • Per-partition lag (uniform vs skewed)
  • Consumer-group state history (rebalance frequency)
  • Sink write latency and throughput for the same window
  • DLQ depth and any stuck committed offsets

Rollback

If a consumer or sink deploy preceded the lag, roll it back; scaling consumers is a mitigation, not a rollback.

Follow-up

  • Switch the alert to lag slope, not absolute lag
  • Verify the DLQ path actually catches poison messages

Diagnosis

  • Get per-partition lag: uniform vs skewed tells you capacity vs key skew.
  • Check the consumer-group state history for `PreparingRebalance`.
  • Look for one partition whose committed offset is stuck.

Possible causes

  • Downstream sink (DB/warehouse) is the bottleneck
  • A hot key routes to a single partition
  • Rebalance loop stealing consumption time
  • A poison message retried forever
  • Too few consumers for the partition count

Commands

kafka-consumer-groups --bootstrap-server $B --describe --group $G
kafka-consumer-groups --bootstrap-server $B --describe --group $G --state

Fix

  • Sink-bound → batch/upsert in bulk, or scale the sink.
  • Key skew → re-key the topic or increase partitions (breaks ordering per key).
  • Rebalance → raise session.timeout.ms / max.poll.interval.ms.
  • Poison message → route to a DLQ and commit past it.
  • Never reset offsets to reduce lag; that is silent data loss.

Prevention

  • Alert on lag slope, not on an absolute lag number.
  • A DLQ for messages failing N times.

Monitoring

  • consumer lag per partition, rebalance count, DLQ depth

Common gotchas

  • Never reset offsets to clear lag; that is silent data loss.
  • Uniform lag means capacity; one stuck partition means key skew or a poison message — read per-partition lag before scaling.
  • A rebalance loop can look like slowness: check the group state history for PreparingRebalance.
  • Adding consumers past the partition count buys nothing.

Production considerations

  • Alert on lag slope, not on an absolute lag number.
  • Give messages that fail N times a DLQ instead of an endless retry.

Tradeoffs

Choose this when Costs & caveats
re-keying the topic fixes hot partitions permanently raising partition count breaks per-key ordering while it rebalances

Diagram

consumer lagging
   |
   |- lag uniform --> capacity bound --> scale consumers / sink throughput
   |
   |- one partition stuck --> hot key or poison --> re-key, or DLQ + commit past
   |
   |- PreparingRebalance looping --> rebalance storm --> raise session.timeout.ms

Registry stats

Verification community · Sep 27, 2026
Completeness 75
Trust multiplier ×0.5
Composite 30.25 · adequate
Freshness 27 days since last source check
Difficulty advanced
Severity high
Version 1.3.0
License CC-BY-4.0
Security flags 3