Build
Practical projects
End-to-end starters you can run locally — architecture, sample data, setup steps, tests, failure scenarios and production hardening.
dbt → warehouse → tests → docs
Transform raw landed data into a tested, documented star schema with dbt, and gate every change on data-aware CI.
Details
API → S3 → Spark → Parquet
Land a paginated REST API into S3 raw, transform with PySpark into partitioned Parquet, and make the whole thing idempotent and testable offline.
Details
Pipeline → DQ → quarantine → publish
Put a contract in front of a table: validate each batch, publish the clean rows, quarantine the rest with reasons, and alert on the quarantine rate.
Details
S3 → Glue → Athena
A serverless, query-in-place pattern: land Parquet in S3, catalogue it with Glue, and query with Athena while keeping scanned bytes (and cost) low.
Details
Kafka → Spark Streaming → Delta
A low-latency stream into Delta Lake with watermarking, windowed aggregation and idempotent upserts that survive replays.
Details
Postgres → CDC → Kafka → Lakehouse
Capture changes from Postgres with Debezium, stream through Kafka, and merge into a lakehouse table with correct ordering, deletes and replay safety.