Own the consumer side — offsets, rebalances, and the KIP-848 protocol

At a glance

Learning objectives

• Explain how consumer groups and offsets decide who reads what
• Detect a rebalance loop from consumer-group state history
• Know what the KIP-848 protocol changes about rebalancing

What you need

• kafka-consumer-groups CLI access to the cluster
• [object Object]

Steps

1. Ground yourself in the mechanics — consumer groups and offsets are what decide who reads what, and delivery semantics decide what a restart means. (from ref-kafka-docs )
2. Get per-partition lag — uniform vs skewed tells you capacity vs key skew. (from runbook-kafka-consumer-lagging )
3. Check the consumer-group state history for `PreparingRebalance`. (from runbook-kafka-consumer-lagging )
4. Read the signature — lag resets to zero then climbs means a rebalance loop; check `session.timeout.ms`. (from skill-kafka-lag-triage )
5. Know where the protocol is going — the KIP-848 consumer protocol in Apache Kafka 4.0 redesigns consumer group rebalancing to eliminate stop-the-world pauses. (from ref-optimize-consumer-rebalancing-on-amazon-msk-with-next-generation )
6. When a rebalance loop is the bug — raise session.timeout.ms / max.poll.interval.ms. (from runbook-kafka-consumer-lagging )
7. Before scaling the fleet — adding consumers past the partition count buys nothing. (from runbook-kafka-consumer-lagging )

Check yourself

• Frequent `PreparingRebalance` in the group state history, with lag that resets to zero then climbs, is a rebalance loop — not a slow consumer.
• A 13th consumer on a 12-partition topic changes nothing — partitions cap parallelism.

Worked example

kafka-consumer-groups --bootstrap-server $B --describe --group $G
kafka-consumer-groups --bootstrap-server $B --describe --group $G --state

- Lag resets to zero then climbs → rebalance loop; check `session.timeout.ms`.
- Is the consumer group stable? Frequent `PreparingRebalance` is the real bug.

The KIP-848 consumer protocol in Apache Kafka 4.0 redesigns consumer
group rebalancing to eliminate stop-the-world pauses.

Sourced from

• Apache Kafka Documentation
• Kafka consumer lagging
• Kafka Consumer Lag Triage
• Optimize consumer rebalancing on Amazon MSK with next generation protocol

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• A rebalance loop can look like slowness: check the group state history for PreparingRebalance.
• Adding consumers past the partition count buys nothing.
• Stop-the-world pauses are the problem the next-generation protocol eliminates — older protocols pause all consumption during a rebalance.

Production considerations

• Alert on lag slope, not on an absolute lag number.
• Rebalance count belongs in monitoring next to consumer lag per partition and DLQ depth.

Diagram

consumer reliability
   |
   |- group subscribes --> partitions split across members
   |
   |- offsets committed per partition --> restart resumes, not re-reads
   |
   |- PreparingRebalance looping --> raise session.timeout.ms / max.poll.interval.ms
   |
   |- consumers > partitions --> idle members, no gain
   |
    --> KIP-848 (Kafka 4.0): rebalancing without stop-the-world pauses

Registry stats