DE//AGENT
Explore
Project community advanced

Postgres → CDC → Kafka → Lakehouse

Capture changes from Postgres with Debezium, stream through Kafka, and merge into a lakehouse table with correct ordering, deletes and replay safety.

At a glance

Where CDC · tech-postgres · tech-debezium · tech-kafka

Learning objectives

  • Configure a logical replication slot safely
  • Order events by LSN and dedupe before merging
  • Handle tombstones without losing history

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Implementation

  • Debezium with `decimal.handling.mode=string` to avoid precision loss.
  • Read the stream, keep the latest event per key by (lsn, ts_ms).
  • MERGE on the primary key; `op='d'` → soft delete with `_deleted_at`.
  • Checkpoint to object storage so a restart resumes correctly.

Tests

  • replaying the same batch produces an identical table
  • deletes propagate as soft deletes within the SLA
  • no duplicate primary keys after a replay

Failure Scenarios

  • Replication slot dropped / WAL purged → full re-snapshot required
  • Out-of-order events flip a value back → ordering used arrival time
  • Schema change in source breaks the connector → handle and evolve

Production Hardening

  • Alert on slot retained WAL and connector task failures
  • Monitor source-vs-target row counts continuously
  • Backpressure with `maxOffsetsPerTrigger`

Registry stats

Verification community · Sep 25, 2026
Completeness 81
Trust multiplier ×0.25
Composite 33.65 · adequate
Freshness 29 days since last source check
Difficulty advanced
Version 1.1.0
License MIT
Security flags 4

Repository Structure

postgres-cdc-kafka-lakehouse/
  docker-compose.yml        # postgres, kafka, connect, minio, spark
  connect/orders-connector.json
  stream/merge_to_lake.py
  ddl/target_table.sql
  tests/test_merge.py

Architecture

Postgres (WAL) ─▶ Debezium ─▶ Kafka (orders.cdc) ─▶ Spark SS [dedupe by LSN] │ ▼ MERGE into Delta/Iceberg

Setup

  • make up # postgres + kafka + connect + minio
  • make register-connector
  • make seed-mutations # insert/update/delete

Sample Data

A seeded Postgres `orders` table; a script that performs inserts, updates and a delete to exercise all CDC ops.