Apache Kafka

DE//AGENT currently lists 5 resources under Apache Kafka: 2 templates, 2 projects, 1 skill. Everything below is derived from the live registry — counts, capabilities, verification state and security posture come from the curated records, never from marketing copy.

Documented learning objectives across these resources:

• Configure a logical replication slot safely — via Postgres → CDC → Kafka → Lakehouse
• Order events by LSN and dedupe before merging — via Postgres → CDC → Kafka → Lakehouse
• Handle tombstones without losing history — via Postgres → CDC → Kafka → Lakehouse
• Read per-partition lag to separate skew from throughput — via Kafka Consumer Lag Triage
• Detect a rebalance loop from consumer-group state — via Kafka Consumer Lag Triage
• Find the poison message from retry counts — via Kafka Consumer Lag Triage
• Choose event-time vs processing-time windows — via Kafka → Spark Streaming → Delta
• Set a watermark and reason about late data — via Kafka → Spark Streaming → Delta

1 more objectives live on the individual resource pages.

These resources map to stages: Streaming (4), Cloud (1). Follow the four-phase roadmap to place this topic inside a progression instead of learning it in isolation.

Every entry, its shape and its trust state — sorted by the registry's composite score. This table is the honest version of a marketing comparison: no sponsored placements, no unverifiable claims.

How scores are computed: source quality score.

Highest composite score first:

1. Local Docker Data Stack score 34.1 — beginner
2. Postgres → CDC → Kafka → Lakehouse score 33.65 — advanced
3. Kafka Consumer Lag Triage score 33.5 — advanced

Scores rank provenance, completeness and trust — read how verification works before adopting anything.

Start here if you're new: Local Docker Data Stack.

Reach for these with production experience: Postgres → CDC → Kafka → Lakehouse, Kafka Consumer Lag Triage, Kafka → Spark Streaming → Delta.

Security-flagged entries: Local Docker Data Stack (2 flags), Postgres → CDC → Kafka → Lakehouse (4 flags), Kafka Consumer Lag Triage (3 flags), Kafka → Spark Streaming → Delta (4 flags), Kafka Streaming Job Template (4 flags). Flags mean the record declares capabilities like credential use or write access — read the security posture on each page before connecting.

• Local Docker Data Stack — A docker-compose stack (Postgres, Kafka, MinIO, Spark, and a DuckDB CLI) for developing and testing pipelines entirely offline. details
• Postgres → CDC → Kafka → Lakehouse — Capture changes from Postgres with Debezium, stream through Kafka, and merge into a lakehouse table with correct ordering, deletes and replay safety. details
• Kafka Consumer Lag Triage — Triages consumer lag from the consumer-group offsets: distinguishing a slow consumer, a partitioned hot key, a rebalance storm, and a poison-pill retry loop. details
• Kafka → Spark Streaming → Delta — A low-latency stream into Delta Lake with watermarking, windowed aggregation and idempotent upserts that survive replays. details

Real install commands from the registry — read the source before running any of them:

amp template new de-agent/docker-local-stack ./local

Local Docker Data Stack — review first

npx de-agent-platform add kafka-consumer-lag-triage

Kafka Consumer Lag Triage — review first

amp template new de-agent/kafka-streaming ./streaming

Kafka Streaming Job Template — review first

Official source: Apache Kafka (license: Apache-2.0) — start there for the canonical documentation, then come back for the vetted registry entries.

Adjacent topics: Apache Spark (4 shared), Delta Lake (3 shared), PostgreSQL (2 shared), template (2 shared).

Study path: the roadmap places this topic in a progression · the scoring model explains the numbers · the trust rule catalog shows every security check these entries passed through.