DE//AGENT
Explore
Project community advanced

Kafka → Spark Streaming → Delta

A low-latency stream into Delta Lake with watermarking, windowed aggregation and idempotent upserts that survive replays.

At a glance

Where Streaming · tech-kafka · tech-spark · tech-delta

Learning objectives

  • Choose event-time vs processing-time windows
  • Set a watermark and reason about late data
  • Make the sink idempotent under replay

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Implementation

  • Read with `startingOffsets=earliest` in dev only; latest in prod.
  • Dedupe on `event_id` within the watermark.
  • Windowed aggregation to 5-minute tumbling windows, appended idempotently.
  • MERGE aggregates on the window key so replays converge.

Tests

  • a late event beyond the watermark is dropped (documented, not silent)
  • replaying a batch yields the same aggregate

Failure Scenarios

  • Checkpoint lost → duplicates; fix by making the sink idempotent
  • Skewed key → one partition saturates; re-key
  • Watermark too short → late data silently dropped

Production Hardening

  • Checkpoints on durable storage with lineage
  • Lag + batch-duration metrics; alert on rising batch duration
  • Compaction job on the aggregate table

Registry stats

Verification community · Sep 23, 2026
Completeness 81
Trust multiplier ×0.25
Composite 31.65 · adequate
Freshness 31 days since last source check
Difficulty advanced
Version 1.0.0
License MIT
Security flags 4

Repository Structure

kafka-spark-delta/
  src/stream_raw.py
  src/stream_agg.py
  conf/stream.yml
  tests/test_window.py
  docker-compose.yml

Architecture

Kafka (events) ─▶ Spark SS [watermark 2m, dedupe on event_id] ├─▶ Delta: raw_events (append, bronze) └─▶ Delta: agg_5min (windowed, MERGE on window key)

Setup

  • make up
  • make produce # emits events, some deliberately late
  • make stream &

Sample Data

A producer script emitting synthetic events with a controllable late-arrival fraction.