watermark

DE//AGENT currently lists 3 resources under watermark: 2 tutorials, 1 pattern. Everything below is derived from the live registry — counts, capabilities, verification state and security posture come from the curated records, never from marketing copy.

Documented learning objectives across these resources:

• Bound incremental work with a watermark and a lookback window — via Incremental processing with a watermark
• Write incremental output idempotently so a re-run is safe — via Incremental processing with a watermark
• Absorb late-arriving files without re-running history — via Incremental processing with a watermark
• Choose event-time vs processing-time windows — via Window a stream correctly — watermarks, dedupe, and late data
• Set a watermark and reason about late data — via Window a stream correctly — watermarks, dedupe, and late data
• Run a structured-streaming job with checkpointing and an idempotent MERGE sink — via Window a stream correctly — watermarks, dedupe, and late data

These resources map to stages: Data Processing (2), Streaming (1). Follow the four-phase roadmap to place this topic inside a progression instead of learning it in isolation.

Every entry, its shape and its trust state — sorted by the registry's composite score. This table is the honest version of a marketing comparison: no sponsored placements, no unverifiable claims.

How scores are computed: source quality score.

Highest composite score first:

1. Incremental Batch with a Watermark score 37.25 — intermediate
2. Incremental processing with a watermark score 34.15 — intermediate
3. Window a stream correctly — watermarks, dedupe, and late data score 34.15 — intermediate

Scores rank provenance, completeness and trust — read how verification works before adopting anything.

Known limits: Little history,Frequently mutating data without a reliable change marker (use CDC) (Incremental Batch with a Watermark)

• Incremental processing with a watermark — Full reloads stop scaling once a table grows past millions of rows. Process a bounded window instead: a watermark ends it, a lookback absorbs late files, and an idempotent write (p… details
• Window a stream correctly — watermarks, dedupe, and late data — Stream processing in practice: start from the kafka-streaming template, pick event-time over processing-time windows, set a watermark you can defend, and prove what happens to late… details

Adjacent topics: tutorial (2 shared), batch (2 shared), incremental (2 shared).

Study path: the roadmap places this topic in a progression · the scoring model explains the numbers · the trust rule catalog shows every security check these entries passed through.