Roadmap · stage 7/13

Streaming

Stage 7 of 13 · 27 resources · 3 categories

What this stage is

Streaming is stage 7 of 13 in the DE//AGENT roadmap. It currently holds 27 resources: 10 tutorials, 4 references, 3 patterns across 3 categories.

What it teaches

Event time, windows, watermarks and exactly-once.

What you can do after this stage

Design an event schema
Recover consumer lag without data loss

Exit gate

Gate: You can run a streaming pipeline that survives replay. The stage deliverable: project-kafka-spark-delta.

Gates are self-assessed — the roadmap does not track your progress; it tells you what to be able to do before moving on.

Categories in this stage

Streaming (8), Kafka (13), CDC (6)

Registry resources for this stage

Sorted by composite score — provenance, completeness and trust, per the scoring model:

Resource	Type	Difficulty	Verification	Score
CDC Merge Designer	Skill	advanced	community	39.5
Kafka MCP Server	MCP server	advanced	community	37.6
Lambda vs Kappa (batch + stream)	Pattern	advanced	community	34.75
Postgres CDC to Lakehouse	Pattern	advanced	community	34.25
Hold the streaming mental model: events, partitions, delivery semantics	Tutorial	beginner	community	34.15
Inspect Kafka topics and groups with the Kafka MCP server	Tutorial	beginner	community	34.15
Troubleshoot a lagging group with the Kafka Troubleshooter	Tutorial	beginner	community	34.15
Debug a Kafka stream with the Kafka Console plugin	Tutorial	intermediate	community	34.15
Make the sink idempotent so replays never duplicate rows	Tutorial	intermediate	community	34.15
Own the consumer side — offsets, rebalances, and the KIP-848 protocol	Tutorial	intermediate	community	34.15
Production gotchas — the configs and capacities that break streams	Tutorial	intermediate	community	34.15
Stream changes, not tables — CDC from Postgres into the lakehouse	Tutorial	intermediate	community	34.15
Triage Kafka consumer lag from per-partition offsets	Tutorial	intermediate	community	34.15
Window a stream correctly — watermarks, dedupe, and late data	Tutorial	intermediate	community	34.15
Postgres → CDC → Kafka → Lakehouse	Project	advanced	community	33.65
Kafka Consumer Lag Triage	Skill	advanced	community	33.5
Streaming Dedup and Upsert	Pattern	advanced	community	31.75
Kafka → Spark Streaming → Delta	Project	advanced	community	31.65
Kafka Troubleshooter	Agent	advanced	community	30.5
Kafka consumer lagging	Runbook	advanced	community	30.25
CDC stopped	Runbook	advanced	community	29.25
Kafka Console	Plugin	advanced	community	28.6
Apache Kafka Documentation	Reference	—	community	27.7
Kafka Streaming Job Template	Template	advanced	community	27.1
Debezium Documentation	Reference	—	community	26.7
Daily Life as a Jenkins Administrator	Reference	—	community	23.55
Optimize consumer rebalancing on Amazon MSK with next generation protocol	Reference	—	community	23.55

Documented learning objectives

Order change events deterministically
Handle deletes (tombstones) without losing history
Make the merge idempotent under replay
Explain why ordering and partitions behave the way they do
Treat at-least-once delivery and replays as the normal case
Choose between lambda, kappa, and plain batch for a latency and reprocessing need
Configure the Kafka MCP server against a broker list
Inventory topics and read group lag with the four tools

Continue the progression

← Orchestration · Cloud →

← Full roadmap · All learning topics