Incremental processing with a watermark

At a glance

Learning objectives

• Bound incremental work with a watermark and a lookback window
• Write incremental output idempotently so a re-run is safe
• Absorb late-arriving files without re-running history

What you need

• An append-mostly source with deterministic event or updated timestamps
• A table past millions of rows where full reloads cost too much

Steps

1. Process a window that ends at a watermark and starts a lookback before it: scan from (max(watermark) - lookback) to now. (from pattern-batch-incremental-watermark )
2. The lookback absorbs late arrivals; the watermark bounds the work. (from pattern-batch-incremental-watermark )
3. Write idempotently (partition overwrite or MERGE) so a re-run is safe. (from pattern-batch-incremental-watermark )
4. Handle deletes explicitly (soft-delete or periodic full refresh) — they don't arrive in an append-only stream. (from pattern-batch-incremental-watermark )
5. Watch a naive is_incremental() without a lookback — it silently drops late data. (from pattern-batch-incremental-watermark )
6. Files arrive after the daily job already ran, so yesterday's numbers change the next morning. (from runbook-late-files )
7. Re-process the late window idempotently (partition overwrite or MERGE), and widen the reprocessing window to cover observed lateness. (from runbook-late-files )

Check yourself

• A re-run for the same window produces identical output — partition overwrite or MERGE, no appends.
• Yesterday's late files are absorbed by the lookback, and no full reload ran.

Worked example

Process a window that ends at a watermark and starts a lookback before it.
  scan from (max(watermark) - lookback) to now

The lookback absorbs late arrivals; the watermark bounds the work. Write
idempotently (partition overwrite or MERGE) so a re-run is safe.

aws s3api list-objects-v2 --bucket b --prefix in/ --query 'Contents[].LastModified'

- max file lateness
- count of late arrivals per run

Sourced from

• Incremental Batch with a Watermark
• Late files

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• A naive is_incremental() without a lookback silently drops late data.
• Re-running the whole partition for one late file is wasteful — overwrite windows, not history.
• Timezone mismatch between partner and pipeline shifts lateness by hours — measure arrival times in UTC.

Production considerations

• Adopt a watermark policy: reprocess the last N days every run.
• Use idempotent writes keyed by business date so a re-run is safe.

Diagram

incremental run
   |
   |- watermark --> window = [max(watermark) - lookback, now]
   |
   |- lookback --> late files absorbed, no full reload
   |
   |- write --> OVERWRITE dt=... / MERGE --> re-run safe
   |
   |- deletes --> soft-delete or periodic full refresh

Registry stats