Hold the streaming mental model: events, partitions, delivery semantics

At a glance

Learning objectives

• Explain why ordering and partitions behave the way they do
• Treat at-least-once delivery and replays as the normal case
• Choose between lambda, kappa, and plain batch for a latency and reprocessing need

What you need

• No cluster required — this is a concepts lesson
• The Apache Kafka documentation open beside you

Steps

1. Read the map first — the docs cover architecture and design, delivery semantics, the protocol, consumer groups and offsets, operations, and security. (from ref-kafka-docs )
2. The design section explains why ordering and partitions behave the way they do; the operations section is the runbook companion. (from ref-kafka-docs )
3. Set your expectations for delivery — at-least-once delivery is the norm, replays happen after failure, and consumers must not see duplicates. (from pattern-streaming-dedup-upsert )
4. Name the core tradeoff — streaming is fast but hard; batch is simple but slow; you need both speed and correctness. (from pattern-lambda-vs-kappa )
5. Learn the two architectures — Lambda runs a batch layer (authoritative, reprocessable) plus a speed layer (approximate, low-latency), merged at query time; Kappa is one streaming path where reprocessing means replaying the log, which requires long log retention and replay safety. (from pattern-lambda-vs-kappa )
6. Carry one health probe into every cluster — is lag rising, flat, or falling? Flat at a high number is fine; rising is not. (from skill-kafka-lag-triage )

Check yourself

• You can say why a partition gives per-key ordering and why the design section of the docs is where that behavior is explained.
• Asked whether a nightly batch needs lambda or kappa, you answer with the pattern's own exit — neither, if a nightly batch genuinely meets the SLA.

Worked example

Lambda:  source ─┬─ batch (authoritative) ─┐
                 └─ speed (approx)       ─┴─▶ serve
Kappa:   source ─▶ stream ─▶ table ─▶ serve   (reprocess = replay)

- At-least-once delivery is the norm
- Replays happen after failure
- Consumers must not see duplicates

1. Is lag rising, flat, or falling? Flat at a high number is fine; rising is not.
2. Is lag even across partitions? Skew means a hot key, not capacity.
3. Is the consumer group stable? Frequent `PreparingRebalance` is the real bug.

Sourced from

• Apache Kafka Documentation
• Lambda vs Kappa (batch + stream)
• Streaming Dedup and Upsert
• Kafka Consumer Lag Triage

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• At-least-once is the norm — replays happen after failure, so consumers must tolerate duplicates from day one.
• Flat lag at a high number is fine; rising is not — do not alert on an absolute lag number.
• Lambda means two code paths and two bugs — the speed layer must stay consistent with the batch layer.

Production considerations

• Keep raw immutable so history is always reprocessable — the modern default is a single lakehouse path with replayable streams, a pragmatic kappa.
• Neither pattern if a nightly batch genuinely meets the SLA — keep it simple.

Diagram

streaming mental model
   |
   |- events, not rows --> append-only log, ordered per partition
   |
   |- at-least-once --> replays happen after failure --> consumers dedupe
   |
   |- batch: simple + slow | stream: fast + hard
   |
   |- lambda --> batch (authoritative) + speed (approx) --> serve
   |- kappa  --> one path; reprocess = replay the log
   |
    --> probe every cluster: is lag rising, flat, or falling?

Registry stats