Stream changes, not tables — CDC from Postgres into the lakehouse

At a glance

Learning objectives

• Read the write-ahead log instead of polling source tables
• Order events by LSN and dedupe before merging
• Handle tombstones without losing history

What you need

• A Postgres source with logical replication available
• [object Object]

Steps

1. Name the gap batch extraction leaves — it loads the data and still misses deletes and mid-day changes. (from pattern-postgres-cdc-lakehouse )
2. Read the write-ahead log, not the tables — WAL → Debezium → Kafka → [dedupe by LSN] → MERGE into lakehouse. (from pattern-postgres-cdc-lakehouse )
3. Configure Debezium with `decimal.handling.mode=string` to avoid precision loss. (from project-postgres-cdc-kafka-lakehouse )
4. Read the stream, keep the latest event per key by (lsn, ts_ms); MERGE on the primary key; `op='d'` → soft delete with `_deleted_at`. (from project-postgres-cdc-kafka-lakehouse )
5. Treat deletes as events — a delete is an event with `op = 'd'`; choose hard delete on merge, or soft delete + `_deleted_at` (required if you need history or audit). (from skill-cdc-merge-designer )
6. Configure a logical replication slot safely. (from project-postgres-cdc-kafka-lakehouse )
7. Know the watch item before going live — a dropped replication slot forces a full re-snapshot. (from pattern-postgres-cdc-lakehouse )

Check yourself

• A row deleted in Postgres shows up in the lakehouse as an event with `op = 'd'` — applied as a hard delete or a soft delete with `_deleted_at`, not as a missing row.
• Replaying the same batch produces an identical table, because events are ranked by (lsn, ts_ms) and deduped before the merge.

Worked example

make up  # postgres + kafka + connect + minio
make register-connector
make seed-mutations  # insert/update/delete

SELECT slot_name, active, restart_lsn, confirmed_flush_lsn FROM pg_replication_slots;

Postgres WAL ─▶ Debezium ─▶ Kafka topic ─▶ stream dedupe ─▶ MERGE ─▶ Delta/Iceberg

Sourced from

• Postgres CDC to Lakehouse
• Postgres → CDC → Kafka → Lakehouse
• CDC Merge Designer
• CDC stopped

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Replication slot dropped / WAL purged → full re-snapshot required.
• Out-of-order events flip a value back → ordering used arrival time.
• Never read the source tables on a schedule.

Production considerations

• Alert on slot retained WAL and connector task failures.
• Monitor source-vs-target row counts continuously.

Diagram

postgres cdc pipeline
   |
   |- Postgres WAL --> Debezium --> Kafka topic
   |
   |- dedupe by LSN --> latest event per key
   |
   |- MERGE on primary key; op='d' --> soft delete (_deleted_at)
   |
    --> Delta/Iceberg; alert on slot retained WAL + task failures

Registry stats