Triage Kafka consumer lag from per-partition offsets

At a glance

Learning objectives

• Read per-partition lag to separate skew from throughput
• Detect a rebalance loop from consumer-group state
• Find the poison message from retry counts

What you need

• kafka-consumer-groups CLI access to the cluster
• The consumer group's describe output and state
• [object Object]

Steps

1. Ask the three questions — is lag rising, flat, or falling? Is it even across partitions? Is the consumer group stable? (from skill-kafka-lag-triage )
2. Get per-partition lag — uniform vs skewed tells you capacity vs key skew. (from runbook-kafka-consumer-lagging )
3. Check the consumer-group state history for `PreparingRebalance`. (from runbook-kafka-consumer-lagging )
4. Look for one partition whose committed offset is stuck. (from runbook-kafka-consumer-lagging )
5. Match the signature — lag on one partition only is key skew; all partitions rising with low CPU is a downstream bottleneck; lag resets to zero then climbs is a rebalance loop; one offset stuck while others advance is a poison message. (from skill-kafka-lag-triage )
6. Fix accordingly — sink-bound: batch/upsert in bulk or scale the sink; key skew: re-key the topic or increase partitions; rebalance: raise session.timeout.ms / max.poll.interval.ms; poison message: route to a DLQ and commit past it. (from runbook-kafka-consumer-lagging )

Check yourself

• Lag confined to two partitions with the rest flat is diagnosed as key skew and fixed by re-keying — not by scaling the fleet.
• Offsets are never reset during the incident; a message that keeps failing lands in a DLQ and the group commits past it.

Worked example

kafka-consumer-groups --bootstrap-server $B --describe --group $G
kafka-consumer-groups --bootstrap-server $B --describe --group $G --state

- Lag on one partition only → key skew; re-key or increase partitions.
- All partitions rising, CPU low → downstream (DB/API) is the bottleneck.
- Lag resets to zero then climbs → rebalance loop; check `session.timeout.ms`.
- One offset stuck, others advance → poison message in a retry loop.

- Sink-bound → batch/upsert in bulk, or scale the sink.
- Key skew → re-key the topic or increase partitions (breaks ordering per key).
- Rebalance → raise session.timeout.ms / max.poll.interval.ms.
- Poison message → route to a DLQ and commit past it.
- Never reset offsets to reduce lag; that is silent data loss.

Sourced from

• Kafka Consumer Lag Triage
• Kafka consumer lagging

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Never reset offsets to reduce lag; that is silent data loss.
• Uniform lag means capacity; one stuck partition means key skew or a poison message — read per-partition lag before scaling.
• A rebalance loop can look like slowness: check the group state history for PreparingRebalance.

Production considerations

• Alert on lag slope, not on an absolute lag number.
• Give messages that fail N times a DLQ instead of an endless retry.

Diagram

consumer lag triage
   |
   |- lag rising? uniform? group stable?
   |
   |- lag uniform --> capacity bound --> scale consumers / sink throughput
   |
   |- one partition stuck --> hot key or poison --> re-key, or DLQ + commit past
   |
   |- PreparingRebalance looping --> rebalance storm --> raise session.timeout.ms
   |
    --> never reset offsets

Registry stats