Delta Lake

DE//AGENT currently lists 5 resources under Delta Lake: 2 skills, 2 projects, 1 template. Everything below is derived from the live registry — counts, capabilities, verification state and security posture come from the curated records, never from marketing copy.

Documented learning objectives across these resources:

• Order change events deterministically — via CDC Merge Designer
• Handle deletes (tombstones) without losing history — via CDC Merge Designer
• Make the merge idempotent under replay — via CDC Merge Designer
• Decide if a table needs OPTIMIZE from file stats — via Delta Optimization Advisor
• Choose Z-ORDER columns from query predicates — via Delta Optimization Advisor
• Know when liquid clustering beats partitioning — via Delta Optimization Advisor
• Configure a logical replication slot safely — via Postgres → CDC → Kafka → Lakehouse
• Order events by LSN and dedupe before merging — via Postgres → CDC → Kafka → Lakehouse

4 more objectives live on the individual resource pages.

These resources map to stages: Lakehouse (1), Streaming (4). Follow the four-phase roadmap to place this topic inside a progression instead of learning it in isolation.

Every entry, its shape and its trust state — sorted by the registry's composite score. This table is the honest version of a marketing comparison: no sponsored placements, no unverifiable claims.

How scores are computed: source quality score.

Highest composite score first:

1. CDC Merge Designer score 39.5 — advanced
2. Delta Optimization Advisor score 34.9 — intermediate
3. Postgres → CDC → Kafka → Lakehouse score 33.65 — advanced

Scores rank provenance, completeness and trust — read how verification works before adopting anything.

Reach for these with production experience: CDC Merge Designer, Postgres → CDC → Kafka → Lakehouse, Kafka → Spark Streaming → Delta.

Security-flagged entries: CDC Merge Designer (1 flag), Delta Optimization Advisor (2 flags), Postgres → CDC → Kafka → Lakehouse (4 flags), Kafka → Spark Streaming → Delta (4 flags), Kafka Streaming Job Template (4 flags). Flags mean the record declares capabilities like credential use or write access — read the security posture on each page before connecting.

• CDC Merge Designer — Produces a MERGE (or Delta/Iceberg merge) that applies inserts, updates and deletes from a CDC stream correctly, including out-of-order events and tombstones. details
• Delta Optimization Advisor — Reads Delta table history and file statistics to decide between OPTIMIZE, Z-ORDER, liquid clustering and partitioning changes. details
• Postgres → CDC → Kafka → Lakehouse — Capture changes from Postgres with Debezium, stream through Kafka, and merge into a lakehouse table with correct ordering, deletes and replay safety. details
• Kafka → Spark Streaming → Delta — A low-latency stream into Delta Lake with watermarking, windowed aggregation and idempotent upserts that survive replays. details

Real install commands from the registry — read the source before running any of them:

npx de-agent-platform add cdc-merge-designer

CDC Merge Designer — review first

npx de-agent-platform add delta-optimization-advisor

Delta Optimization Advisor — review first

amp template new de-agent/kafka-streaming ./streaming

Kafka Streaming Job Template — review first

Official source: Delta Lake (license: Apache-2.0) — start there for the canonical documentation, then come back for the vetted registry entries.

Adjacent topics: Apache Spark (3 shared), Apache Kafka (3 shared), Apache Iceberg (2 shared), Debezium (2 shared).

Study path: the roadmap places this topic in a progression · the scoring model explains the numbers · the trust rule catalog shows every security check these entries passed through.