Learn the pattern.Install the skill.Run the pipeline.Ship agent-native data.

Filesystem MCP Server

Reference server that exposes a sandboxed set of directories as readable and (optionally) writable tools.

Git MCP Server

Exposes local Git operations — status, diff, log, commit, branch — as tools, so an agent can inspect history and prepare changes without shelling out blindly.

Fetch MCP Server

Retrieves a URL and converts it to markdown for the model.

dbt → warehouse → tests → docs

Transform raw landed data into a tested, documented star schema with dbt, and gate every change on data-aware CI.

API → S3 → Spark → Parquet

Land a paginated REST API into S3 raw, transform with PySpark into partitioned Parquet, and make the whole thing idempotent and testable offline.

Pipeline → DQ → quarantine → publish

Put a contract in front of a table: validate each batch, publish the clean rows, quarantine the rest with reasons, and alert on the quarantine rate.

S3 → Glue → Athena

A serverless, query-in-place pattern: land Parquet in S3, catalogue it with Glue, and query with Athena while keeping scanned bytes (and cost) low.

Kafka → Spark Streaming → Delta

A low-latency stream into Delta Lake with watermarking, windowed aggregation and idempotent upserts that survive replays.

Postgres → CDC → Kafka → Lakehouse

Capture changes from Postgres with Debezium, stream through Kafka, and merge into a lakehouse table with correct ordering, deletes and replay safety.

Catalogs and ownership: who owns this table and is it safe to use?

Every incident resolves faster when you can ask a catalog what this data is, who owns it, and whether it is safe to use.

Express a freshness check as a metric and an alert

The first hands-on of the series: declare a freshness SLA, express staleness as a metric, and write an alert rule — including what happens when no data arrives at all.

Hold the observability mental model: infrastructure metrics versus data signals

Two layers live on one data platform: infrastructure metrics (CPU, lag, job success) and data signals (freshness, volume, schema, distribution).

Hold the row-versus-column mental model

Before you pick a file format, understand why analytics engines read columnar files at all: row groups carrying min/max statistics let a reader prune most of the file before decoding it, and columnar layout means a scan of three columns never touches the other twenty.

Hold the system design mental model — data systems as tradeoffs

• Frame any data system as a tradeoff among reliability, scalability, and maintainability
• Name the six mechanisms DDIA teaches and map each one to a system you operate

Open the Parquet file — row groups, pages, encodings

A Parquet file is not a black box — it is row groups of column chunks made of pages, with statistics that readers use for predicate pushdown.