Series · roadmap stage 7/13

The Streaming Series

Ten lessons in order, from the mental model to a capstone checklist. Every lesson is structured by us and grounded in a real, cited source from the registry — no invented commands.

10 of 10 lessons published

1. Lesson 1 Hold the streaming mental model: events, partitions, delivery semantics

   Build the vocabulary before touching a cluster: events instead of rows, partitions instead of tables, at-least-once delivery instead of exactly-once, and the lambda/kappa decision for speed versus correctness. Every later lesson in this ser

2. Lesson 2 Inspect Kafka topics and groups with the Kafka MCP server

   Connect an agent to a Kafka cluster through the Kafka MCP server and work through its four tools: list_topics for inventory, describe_topic for a single topic, describe_consumer_group for lag, and consume_messages for sampling — stream debu

3. Lesson 3 Make the sink idempotent so replays never duplicate rows

   The core producer-side reliability skill for streams: you cannot stop replays, so the sink must converge. Deduplicate by a business key and event time within the watermark, MERGE on the key, and rank change events by (lsn, ts_ms) — never by

4. Lesson 4 Own the consumer side — offsets, rebalances, and the KIP-848 protocol

   Consumer reliability is where streams quietly break: a rebalance loop looks like slowness, consumers past the partition count buy nothing, and stop-the-world rebalances pause consumption. Read group state, fix the timeouts, and know what th

5. Lesson 5 Window a stream correctly — watermarks, dedupe, and late data

   Stream processing in practice: start from the kafka-streaming template, pick event-time over processing-time windows, set a watermark you can defend, and prove what happens to late data. Checkpointing and an idempotent MERGE sink make the j

6. Lesson 6 Stream changes, not tables — CDC from Postgres into the lakehouse

   The intermediate pattern that replaces nightly extraction: read the write-ahead log through Debezium, stream through Kafka, dedupe by LSN, and MERGE into Delta/Iceberg. Covers tombstones, decimal handling, and what a dropped replication slo

7. Lesson 7 Debug a Kafka stream with the Kafka Console plugin

   Drive the Kafka Console plugin's three commands across a lag incident: /topic-info for inventory, /lag-triage for per-partition lag and rebalance detection, and /sample-topic for a read-only message sample with a PII redaction pass. Permiss

8. Lesson 8 Production gotchas — the configs and capacities that break streams

   The five causes of consumer lag on one card, the two destructive shortcuts that must never happen, and the capacity trap of scaling past the partition count. Ends with the alert that matters — lag slope, not absolute lag — and the sampling

9. Lesson 9 Triage Kafka consumer lag from per-partition offsets

   Combine the lag-triage skill's three questions with the lagging-consumer runbook: read per-partition lag to separate capacity from key skew, check the group state history for a rebalance loop, find the one stuck offset that marks a poison m

10. Lesson 10 Troubleshoot a lagging group with the Kafka Troubleshooter

    Hand the Kafka Troubleshooter agent the consumer-group describe output, per-partition offsets, and consumer logs, then follow its four-step workflow from lag trend to the least destructive fix. The agent works read-only against the cluster,

← Streaming stage · All tutorials