Incremental Batch with a Watermark

At a glance

Problem

Full reloads are too slow and too expensive once a table grows past millions of rows.

Solution

Process a window that ends at a watermark and starts a lookback before it. scan from (max(watermark) - lookback) to now The lookback absorbs late arrivals; the watermark bounds the work. Write idempotently (partition overwrite or MERGE) so a re-run is safe. Handle deletes explicitly (soft-delete or periodic full refresh) — they don't arrive in an append-only stream.

When not to use

• Little history
• Frequently mutating data without a reliable change marker (use CDC)

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Context

A batch pipeline reading an append-mostly source that has an updated_at/event time.

Forces

• Cost grows with history size
• Late-arriving rows must not be lost
• Re-runs must be safe
• Deletes may never be seen

Tradeoffs

• watch: a naive is_incremental() without a lookback silently drops late data

Diagram

[max(updated_at) - N days] ─────▶ [now]
          lookback              watermark

Registry stats