dedup

DE//AGENT currently lists 2 resources under dedup: 1 tutorial, 1 pattern. Everything below is derived from the live registry — counts, capabilities, verification state and security posture come from the curated records, never from marketing copy.

Documented learning objectives across these resources:

• Make the sink idempotent rather than trying to make the source exactly-once — via Make the sink idempotent so replays never duplicate rows
• Deduplicate by a business key + event time within the watermark — via Make the sink idempotent so replays never duplicate rows
• Rank change events deterministically by (lsn, ts_ms), never by arrival time — via Make the sink idempotent so replays never duplicate rows

These resources map to stage: Streaming (2). Follow the four-phase roadmap to place this topic inside a progression instead of learning it in isolation.

Every entry, its shape and its trust state — sorted by the registry's composite score. This table is the honest version of a marketing comparison: no sponsored placements, no unverifiable claims.

How scores are computed: source quality score.

Highest composite score first:

1. Make the sink idempotent so replays never duplicate rows score 34.15 — intermediate
2. Streaming Dedup and Upsert score 31.75 — advanced

Scores rank provenance, completeness and trust — read how verification works before adopting anything.

Reach for these with production experience: Streaming Dedup and Upsert.

Security-flagged entries: Streaming Dedup and Upsert (2 flags). Flags mean the record declares capabilities like credential use or write access — read the security posture on each page before connecting.

Known limits: Pure append-only event log with no dedup need (Streaming Dedup and Upsert)

• Make the sink idempotent so replays never duplicate rows — The core producer-side reliability skill for streams: you cannot stop replays, so the sink must converge. Deduplicate by a business key and event time within the watermark, MERGE o… details

Adjacent topics: streaming (2 shared).

Study path: the roadmap places this topic in a progression · the scoring model explains the numbers · the trust rule catalog shows every security check these entries passed through.