Window a stream correctly — watermarks, dedupe, and late data

At a glance

Learning objectives

• Choose event-time vs processing-time windows
• Set a watermark and reason about late data
• Run a structured-streaming job with checkpointing and an idempotent MERGE sink

What you need

• The kafka-streaming template (checkpointing, watermarking, a dedupe stage, an idempotent MERGE sink)
• [object Object]
• A Kafka topic whose events carry an event timestamp and an event_id

Steps

1. Start from the template — a structured-streaming job with checkpointing, watermarking, a dedupe stage and an idempotent MERGE sink to Delta/Iceberg. (from tpl-kafka-streaming )
2. Decide time semantics first — choose event-time vs processing-time windows. (from project-kafka-spark-delta )
3. Follow the reference architecture — Spark SS with watermark 2m and dedupe on event_id, raw_events appended to bronze, agg_5min as a windowed MERGE on the window key. (from project-kafka-spark-delta )
4. Read with `startingOffsets=earliest` in dev only; latest in prod. (from project-kafka-spark-delta )
5. Dedupe on `event_id` within the watermark; windowed aggregation to 5-minute tumbling windows, appended idempotently. (from project-kafka-spark-delta )
6. Prove late-data behavior in tests — a late event beyond the watermark is dropped (documented, not silent), and replaying a batch yields the same aggregate. (from project-kafka-spark-delta )
7. Watch the two classic failures — watermark too short means late data silently dropped; checkpoint lost means duplicates, fixed by making the sink idempotent. (from project-kafka-spark-delta )

Check yourself

• An event arriving inside the 2-minute watermark joins its window; one beyond it is dropped — documented in the tests, not silent.
• Replaying a batch yields the same aggregate because the MERGE is on the window key.

Worked example

make up
make produce  # emits events, some deliberately late
make stream &

- Read with `startingOffsets=earliest` in dev only; latest in prod.
- Dedupe on `event_id` within the watermark.
- Windowed aggregation to 5-minute tumbling windows, appended idempotently.
- MERGE aggregates on the window key so replays converge.

Kafka (events) ─▶ Spark SS [watermark 2m, dedupe on event_id]
                  ├─▶ Delta: raw_events (append, bronze)
                  └─▶ Delta: agg_5min  (windowed, MERGE on window key)

Sourced from

• Kafka Streaming Job Template
• Kafka → Spark Streaming → Delta

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Watermark too short → late data silently dropped.
• Read with `startingOffsets=earliest` in dev only; latest in prod.
• Skewed key → one partition saturates; re-key.

Production considerations

• Checkpoints on durable storage with lineage.
• Lag + batch-duration metrics; alert on rising batch duration.

Diagram

windowed streaming job
   |
   |- Kafka events (event_id, event time)
   |
   |- watermark 2m --> dedupe on event_id
   |
   |- 5-minute tumbling windows --> aggregate
   |
   |- raw_events --> append (bronze)
   |- agg_5min  --> MERGE on window key (idempotent)
   |
    --> checkpoint on durable storage; alert on rising batch duration

Registry stats