An open registry and learning platform for modern Data Engineering: tutorials, architecture patterns, agent skills, MCP servers, agents, hooks and production runbooks — each with provenance, license, compatibility and verification status.https://emanisha.in/https://emanisha.in/de-agent/tutorials/obs-catalog-ownership/https://emanisha.in/de-agent/tutorials/obs-catalog-ownership/Every incident resolves faster when you can ask a catalog what this data is, who owns it, and whether it is safe to use. Run the catalog-assistant workflow end to end — resolve, retrieve, summarise — and keep it read-only and metadata-only by design.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/obs-first-freshness-alert/https://emanisha.in/de-agent/tutorials/obs-first-freshness-alert/The first hands-on of the series: declare a freshness SLA, express staleness as a metric, and write an alert rule — including what happens when no data arrives at all. Ends with the pre-merge check that keeps schedules honest against the SLA they claim.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/obs-mental-model/https://emanisha.in/de-agent/tutorials/obs-mental-model/Two layers live on one data platform: infrastructure metrics (CPU, lag, job success) and data signals (freshness, volume, schema, distribution). This lesson separates them, places lineage and catalogs on the map, and names the four data signals every pipeline owes you before you write a dashboard.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/foundations-rows-vs-columns/https://emanisha.in/de-agent/tutorials/foundations-rows-vs-columns/Before you pick a file format, understand why analytics engines read columnar files at all: row groups carrying min/max statistics let a reader prune most of the file before decoding it, and columnar layout means a scan of three columns never touches the other twenty. Every later lesson in this series builds on this model.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/system-design-mental-model/https://emanisha.in/de-agent/tutorials/system-design-mental-model/Before any diagram, data systems are tradeoffs, not products. This lesson fixes the vocabulary — reliability, scalability, maintainability, and the six canonical mechanisms behind them — and shows how the canonical chapters map onto systems you already run, so every later lesson in the series has a place to land.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/foundations-parquet-anatomy/https://emanisha.in/de-agent/tutorials/foundations-parquet-anatomy/A Parquet file is not a black box — it is row groups of column chunks made of pages, with statistics that readers use for predicate pushdown. Learn the anatomy from the two Parquet sources and you can reason about every read and every layout decision later in this series.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/system-design-back-of-envelope/https://emanisha.in/de-agent/tutorials/system-design-back-of-envelope/Estimation is the warm-up act of every design conversation: QPS, storage per day, retention, peak factor, all written down before a single box is drawn. This lesson drills the powers-of-two arithmetic and shows how to ground the numbers in architectures that actually ship.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/foundations-twelve-factor-pipelines/https://emanisha.in/de-agent/tutorials/foundations-twelve-factor-pipelines/Data pipelines run as jobs on shared infrastructure, so the twelve-factor principles apply to them — config in the environment, backing services as attached resources, logs as event streams, disposability, and dev/prod parity. Learn the vocabulary engineers already share when arguing about config, logs, and process design.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/foundations-arrow-zero-copy/https://emanisha.in/de-agent/tutorials/foundations-arrow-zero-copy/Between the file on disk and the engine doing work sits memory. Apache Arrow defines the columnar in-memory format, its language bindings, the IPC and Flight protocols, and zero-copy interchange between systems — which is why modern data tools can hand columns to each other without re-serializing everything in between.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/foundations-avro-schema-evolution/https://emanisha.in/de-agent/tutorials/foundations-avro-schema-evolution/Event payloads change while consumers keep running. Avro's answer is a schema declared next to the data, evolution rules carried by the schema registry, and a container format used across event and stream pipelines. Walk the Avro docs' own four topics and see where the format sits against Parquet and Arrow.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/foundations-capstone-checklist/https://emanisha.in/de-agent/tutorials/foundations-capstone-checklist/Ten lessons compress into one checklist — place every decision on the lifecycle, lay out analytics files with row groups and statistics, expect Arrow at the boundaries, evolve event payloads with Avro's rules, and run jobs twelve-factor style. Each line cites the source that owns the claim.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/obs-capstone-checklist/https://emanisha.in/de-agent/tutorials/obs-capstone-checklist/Ten lessons compress into one page. Declare SLAs, express them as metrics and alerts with explicit no-data handling, instrument runs with traces, emit lineage events, keep the catalog authoritative — and gate both schedule changes and column changes pre-merge.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/system-design-storage-engines/https://emanisha.in/de-agent/tutorials/system-design-storage-engines/Storage choice is where a design's promises become real: the engine you pick decides what replication buys, what partitioning costs, and which consistency semantics your merges get. This lesson builds a decision habit around the DDIA mechanisms instead of brand loyalty.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/foundations-choosing-file-formats/https://emanisha.in/de-agent/tutorials/foundations-choosing-file-formats/The three formats in this series are not competitors — they answer different questions. Parquet is the columnar file for analytics scans, Avro is the schema-evolving container for event payloads, and Arrow is the in-memory substrate between systems. Make the comparison explicit so the next format debate takes minutes, not days.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/obs-column-lineage-concepts/https://emanisha.in/de-agent/tutorials/obs-column-lineage-concepts/Table-level lineage says a job touched a table; column-level lineage says which columns feed which. Learn the concepts, the OpenLineage event model that makes lineage work across tools, and how an impact set is computed from a lineage store.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/foundations-compression-encodings/https://emanisha.in/de-agent/tutorials/foundations-compression-encodings/Encodings and compression codecs are where file size and scan speed are actually decided — Parquet applies encodings and codecs per column chunk with statistics on top, while Avro carries schema plus encoding for event payloads. Learn what each layer buys you before you pick defaults.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/system-design-sli-slo/https://emanisha.in/de-agent/tutorials/system-design-sli-slo/A design without reliability numbers is a wish. This lesson borrows the SRE toolkit — SLIs, SLOs, error budgets, incident management, postmortems, and capacity planning — and points it at data platforms, where freshness and completeness are the properties users actually feel.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/system-design-resilience-patterns/https://emanisha.in/de-agent/tutorials/system-design-resilience-patterns/Failures are inputs, not exceptions. Microsoft's pattern catalogue gives each failure shape a name and a solution — retry, circuit breaker, compensating transaction, bulkhead — and each pattern page carries the considerations that decide when it backfires. This lesson fits them to pipeline components.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/foundations-format-performance-incidents/https://emanisha.in/de-agent/tutorials/foundations-format-performance-incidents/When a scan reads the whole lake for one column, the cause is usually the file layout, not the engine — missing statistics, row groups sized without intent, encodings chosen by default. Walk two incident patterns grounded in what the Parquet sources actually claim about statistics and pruning.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/obs-lineage-incident/https://emanisha.in/de-agent/tutorials/obs-lineage-incident/The source job quietly changed from hourly to daily; the mart's 6h freshness SLA now fails by construction. Run the drill — confirm the breach from the diff, walk upstream with get_lineage, and report the impact set with owner, criticality and the earliest SLA that breaks.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/obs-otel-traces/https://emanisha.in/de-agent/tutorials/obs-otel-traces/Traces give every pipeline run a story — which task ran, how long, with what attributes. Wrap a job with OpenTelemetry, adopt the semantic conventions so attribute names match across teams, and see exactly where runtime traces end and lineage events begin.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/obs-lineage-tooling/https://emanisha.in/de-agent/tutorials/obs-lineage-tooling/Put lineage to work with two tools — a DataHub MCP server that lets an agent search entities and read lineage and schema, and a lineage-graph plugin that turns column-level impact analysis into agent commands. Wire both, learn each tool surface, and scope the token before sharing it.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/obs-prod-gotchas/https://emanisha.in/de-agent/tutorials/obs-prod-gotchas/Three failure modes of a monitoring stack that looks healthy: alerts that page until nobody reads them, metrics with no declared SLA behind them, and no-data windows that pass as health. Grounded in the alerting semantics of Prometheus and Grafana and the schedule-vs-SLA contract.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/system-design-reference-architectures/https://emanisha.in/de-agent/tutorials/system-design-reference-architectures/Reference architectures are precedent, not homework. This lesson teaches a reading method: name the moving parts, find the pillar each choice defends, spot the serverless data lake pattern, and re-state the diagram in DDIA vocabulary so vendor drawings become reusable design evidence.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/foundations-reading-data-files/https://emanisha.in/de-agent/tutorials/foundations-reading-data-files/A read is a pipeline in miniature — the reader consults statistics, prunes row groups, decodes only the columns you asked for, and hands you an in-memory columnar structure. Walk the read path with the sources, then place it on the lifecycle between storage and transformation.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/system-design-scaling-vocabulary/https://emanisha.in/de-agent/tutorials/system-design-scaling-vocabulary/Scale arguments go wrong when the words do: horizontal versus vertical, cache-aside versus write-through, shard by range versus hash. This lesson fixes the Primer's vocabulary and lines it up with DDIA's partitioning and replication so one argument covers both worlds.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/obs-lineage-explainer-walkthrough/https://emanisha.in/de-agent/tutorials/obs-lineage-explainer-walkthrough/Install the lineage-explainer skill and walk it end to end: ask what breaks if a column changes, read technical and business lineage from the store, and produce an impact report with owner, criticality and the earliest SLA that breaks.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/system-design-capstone-checklist/https://emanisha.in/de-agent/tutorials/system-design-capstone-checklist/Nine lessons compress into one checklist: tradeoffs named, numbers written, storage chosen by mechanism, scaling lever identified, budget defined, failure patterns assigned, precedent cited. Run it on your next design and on the platform you already run.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/system-design-interview-walkthrough/https://emanisha.in/de-agent/tutorials/system-design-interview-walkthrough/The full arc in one 45-minute shape: requirements, back-of-envelope numbers, the storage and scaling argument, reliability as SLIs and budgets, and a deployed precedent to close. Every claim traces to one of the five sources, so nothing in the answer is invented under pressure.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/system-design-prod-gotchas/https://emanisha.in/de-agent/tutorials/system-design-prod-gotchas/The two self-inflicted design failures: complexity bought before load requires it, and capacity treated as an afterthought until the pager goes off. This lesson puts the SRE discipline of capacity planning next to the pattern catalogue's warnings, so design reviews catch both before production does.Sat, 24 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/sql-first-correct-select/https://emanisha.in/de-agent/tutorials/sql-first-correct-select/Stand up a local DuckDB through the DuckDB MCP server, inspect before you query with list_tables and describe_table, and write one SELECT with a join whose cardinality you can justify. You read CSV and Parquet straight into the engine, state the grain of the result, and finish by proving the join cannot fan out — the habit the sql-review skill demands.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/cloud-mental-model/https://emanisha.in/de-agent/tutorials/cloud-mental-model/Build the map before opening a console: object storage holds the bytes, a managed catalog describes them, and serverless compute bills for what a query scans. Walk the same shape through five managed engines — Glue, Athena, BigQuery, Snowflake, Databricks — and note the IAM surface each one needs before the first query.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/reliability-mental-model/https://emanisha.in/de-agent/tutorials/reliability-mental-model/Build the map before writing a single check: a consumer experiences a table as three properties — freshness, completeness, correctness — declared as an SLA in a data contract, enforced by a CI test, and spent like an error budget when failures land.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/lakehouse-mental-model/https://emanisha.in/de-agent/tutorials/lakehouse-mental-model/Build the map before touching a table: object storage is files with no structure for trust or ownership, a table format (Iceberg or Delta) adds snapshots, manifests, hidden partitioning, ACID, and evolution, and medallion zones make trust legible from raw to serving. Multi-catalog means the formats coexist — pick each one deliberately.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/orchestration-mental-model/https://emanisha.in/de-agent/tutorials/orchestration-mental-model/Build the map before touching code: a DAG plus its schedule decides when runs happen, the idempotency test decides whether re-runs are safe, catchup decides what happens to missed intervals, and a backfill decides how history gets fixed — four decisions every later lesson sharpens.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/sql-relational-mental-model/https://emanisha.in/de-agent/tutorials/sql-relational-mental-model/Before the first SELECT, build the model every later lesson uses: a query returns a set with a stated grain, a join changes that grain by its cardinality, and NULL never compares as you expect. You learn the first question every SQL review asks, the five rules that follow from it, and the failure modes — fan-out joins, NULL-unsafe predicates, implicit casts, non-deterministic ordering, functions that silently change grain — the rest of the series attacks.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/warehouse-mental-model/https://emanisha.in/de-agent/tutorials/warehouse-mental-model/Build the map before tuning anything: a table is a set of micro-partitions, every query is described by bytes scanned, partitions scanned ratio, and queue time, a clustering key decides how much gets pruned, and a slow query is either a scan problem or a compute problem — never both at once.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/agentic-mental-model/https://emanisha.in/de-agent/tutorials/agentic-mental-model/Build the map before touching anything: an agent is a model that reasons, tools that act, a loop that iterates, and guardrails that bound it. Meet a real data-engineering agent from the registry, point at each of the four parts, and see where the guardrails come from — hooks and the human gate.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/cloud-agent-tooling/https://emanisha.in/de-agent/tutorials/cloud-agent-tooling/Wire the registry's cloud tooling into one agent workflow: install the Glue and Athena review skills, bind the AWS MCP server to a least-privilege profile, and compose both skills plus the MCP server into the architecture reviewer. The same pattern covers GCP with the BigQuery server.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/first-lakehouse-table/https://emanisha.in/de-agent/tutorials/first-lakehouse-table/Run the whole loop in one sitting on a scratch table: create and query a first Delta or Iceberg table from the official docs, read its health from the files metadata table with one SQL query, run a first OPTIMIZE or rewrite_data_files, and prove it worked by comparing numFiles and bytes scanned before and after.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/acid-concurrent-writes/https://emanisha.in/de-agent/tutorials/acid-concurrent-writes/What ACID on object storage actually guards, and where it stops: Delta commits transactions on files, but compaction is also a writer, and the two meet in the ingestion window. Learn the schedule rule, the write-path/layout-path separation, and how a guardrail blocks maintenance from running in the wrong window.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/orchestration-capstone/https://emanisha.in/de-agent/tutorials/orchestration-capstone/Close the series by running the full review pass over your own pipeline: the idempotency test per task, partitioned or MERGE writes, deliberate catchup, backoff retries, reschedule sensors, bounded concurrency — and a documented runbook for the day history needs fixing.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/cloud-capstone-checklist/https://emanisha.in/de-agent/tutorials/cloud-capstone-checklist/Close the series by turning nine lessons into one standing checklist: workgroup bytes limits and compaction on storage, sized workers and stable bookmarks on compute, resource monitors and unit-metric alerts on spend, and least-privilege IAM everywhere — then produce the four review outputs as evidence.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/reliability-capstone/https://emanisha.in/de-agent/tutorials/reliability-capstone/Close the series by running the reliability checklist over your own pipeline: a declared and enforced SLA, idempotent tasks with backoff retries, a documented backfill runbook — then drill the incident path from alert to blameless postmortem with dated, owned action items.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/sql-capstone-checklist/https://emanisha.in/de-agent/tutorials/sql-capstone-checklist/Close the series by turning nine lessons into one standing checklist: a grain statement for every SELECT, a cardinality justification for every join, NULL-safe anti-joins, explicit casts, lint in CI, and one decision rule for slow queries. Each line cites the lesson that earned it, and the checklist ends where the review agent ends — with findings that cite severity, location, rule, and fix.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/lakehouse-capstone/https://emanisha.in/de-agent/tutorials/lakehouse-capstone/Close the series by running the full checklist on one table you own: zones with gates, a deliberately chosen format, the three standing monitors, the four-part maintenance plan as a scheduled job, guardrail-enforced windows, and the escalation line that decides when it pages.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/warehouse-capstone-checklist/https://emanisha.in/de-agent/tutorials/warehouse-capstone-checklist/Close the series by auditing one warehouse project against everything it taught: models with tests and docs, layered staging to marts with a declared grain, incremental loads with a lookback, CI on a staging schema, monitored query history, and a repo structure a new joiner can read.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/reliability-dr-replication/https://emanisha.in/de-agent/tutorials/reliability-dr-replication/Disaster recovery for data systems starts before the outage: DDIA's replication and consistency chapters name the mechanisms, and the registry's runbooks name the recovery paths — code rollback for deterministic failures, quarantine-then-retry for data errors, shadow-then-swap for recomputes.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/dynamic-mapping-date-batches/https://emanisha.in/de-agent/tutorials/dynamic-mapping-date-batches/Run one task definition over many date batches with dynamic task mapping — the docs' mapping feature applied the way the backfill material demands: batched ranges, smallest first, a validated sample after each batch.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/cloud-cost-governance/https://emanisha.in/de-agent/tutorials/cloud-cost-governance/Govern cloud spend the way you govern data quality: right-size each warehouse per environment, put a resource monitor between the workload and the invoice, and alert on cost-per-run instead of total spend. Ends with the unit-economics rule that turns a 4x regression into a number, not a surprise.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/agentic-hooks-gates/https://emanisha.in/de-agent/tutorials/agentic-hooks-gates/The model can be talked into anything; a hook is code on the tool path. Walk the four safety hooks every data repo should know — destructive-SQL guard, prod write confirmation, secret scan, backfill safety check — and wire the two the starter plugin ships.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/reliability-idempotent-reruns/https://emanisha.in/de-agent/tutorials/reliability-idempotent-reruns/Retries are inevitable, so a task that appends duplicates rows every time it is cleared: hold each task to the idempotency test, replace its partition atomically instead of appending, and assert the invariant in CI with a test that runs the same date twice.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/warehouse-query-incident/https://emanisha.in/de-agent/tutorials/warehouse-query-incident/Run the slow-query runbook as an incident: confirm the regression, split scan cost from compute contention, pull partitions_scanned and bytes from query history as evidence, apply the matched fix, and escalate on the shared-warehouse SLA before the queue takes other teams down with it.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/cloud-cost-spike-incident/https://emanisha.in/de-agent/tutorials/cloud-cost-spike-incident/An invoice alert or a cost-per-run dashboard spike lands: one job's compute cost quadrupled with no functional change. Work the runbook — diff shuffle write and input bytes run-over-run, rank the four usual suspects, fix per cause, and cap the damage with a resource monitor while you diagnose.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/sql-incident-silent-fanout/https://emanisha.in/de-agent/tutorials/sql-incident-silent-fanout/Row counts are inflated and metrics double-counted after a retry, a fan-out join or a backfill — the incident signature of a silent fan-out. You work the duplicate-records runbook end to end: count duplicates per key with one query, correlate the first duplicate date with a deploy, check whether the write is append or merge/overwrite, then dedupe on the business key and rewrite only the affected partitions.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/agentic-skill-anatomy/https://emanisha.in/de-agent/tutorials/agentic-skill-anatomy/A skill is the unit of reusable agent behavior: one SKILL.md with front-matter, plus directories the agent loads on demand. Dissect a real registry skill file by file, learn the evaluation rubric its objectives set, then install it and compare the layout with your own draft.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/sensors-data-aware-scheduling/https://emanisha.in/de-agent/tutorials/sensors-data-aware-scheduling/Stop guessing when upstream data arrives: add a sensor in reschedule mode so the DAG waits for its input without holding a worker slot, read the scheduling-semantics pages first, and bound the sensor with timeout and concurrency settings like any other task.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/warehouse-clustering-pruning/https://emanisha.in/de-agent/tutorials/warehouse-clustering-pruning/Choose a clustering key from the predicates of your most expensive queries, prove the prune with partitions_scanned versus partitions_total, and fix the disease you actually have — a lost predicate or a low-selectivity key on the scan side, or queue time on the compute side.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/warehouse-star-schema/https://emanisha.in/de-agent/tutorials/warehouse-star-schema/Turn the analytical layer into facts surrounded by conformed dimensions: declare the fact grain in schema.yml, stage and intermediate models below it, SCD2 where history must survive, and scaffold the whole thing from the dbt project template so staging, marts, tests, and CI exist from commit one.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/reliability-safe-backfills/https://emanisha.in/de-agent/tutorials/reliability-safe-backfills/Historical output is wrong or missing, and the fix touches a production table: define correct first, map the blast radius from lineage, then shadow-write in date batches smallest-first, validate a sample per batch, and swap atomically — behind a gate that demands a rollback path.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/sql-production-gotchas/https://emanisha.in/de-agent/tutorials/sql-production-gotchas/The bugs that survive testing compare wrong instead of failing loud: NOT IN over a nullable subquery, a comparison across types left to an implicit cast, and a schema change that silently turns a column to NULLs. You walk the review rules that ban each pattern, then use the schema-drift runbook to tell a hard failure from a silent one and to decide between an explicit cast and a quarantine.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/reliability-prod-gotchas/https://emanisha.in/de-agent/tutorials/reliability-prod-gotchas/The failures that become pages are the ones nobody was watching: a regression hiding in growth, a cron change that silently widens the freshness SLA, and appends that duplicate on retry — cap each with a guard, a computed worst-case lag, and an atomic partition replace.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/cloud-prod-gotchas/https://emanisha.in/de-agent/tutorials/cloud-prod-gotchas/The surprises that turn into invoices: workers sized by habit, a renamed prefix that silently resets a bookmark, thousands of small files, an agent holding an admin role, and a query that scans everything because the predicate missed the partition column. Each gotcha names the metric that catches it early.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/production-dag-gotchas/https://emanisha.in/de-agent/tutorials/production-dag-gotchas/The DAG failures that become pages: unbounded concurrency, missing catchup policy, and a backfill that starves the daily run — cap them with max_active_runs and pools, then wire the monitoring and escalation path the backfill runbook prescribes.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/schema-evolution-time-travel/https://emanisha.in/de-agent/tutorials/schema-evolution-time-travel/Change a table without breaking its readers or its history: the Iceberg spec's schema and partition evolution, Delta time travel, the history metadata table you query to see what happened, and the snapshot-expiry rule that keeps time travel working while storage stays bounded. The immutable raw zone is the floor under all of it.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/iceberg-metadata-mechanics/https://emanisha.in/de-agent/tutorials/iceberg-metadata-mechanics/Open the hood on an Iceberg table: the spec's layers (snapshots, manifests, hidden partitioning), the files and history metadata tables you query with plain SQL, and the manifest list as a maintenance target in its own right. One catalog then spans Iceberg, Delta, Hudi, and Hive.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/sql-window-functions-grain/https://emanisha.in/de-agent/tutorials/sql-window-functions-grain/A window function computes over a set of rows and hands the answer back to every row, so the result keeps its grain — the property that makes a keep-latest dedupe safe and a bare GROUP BY grain-changing. You find the duplicates first with one GROUP BY query, dedupe on the business key keeping the latest by event time, and learn when a window over the same key is a shuffle that can skew or spill.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/dag-dependencies-retries/https://emanisha.in/de-agent/tutorials/dag-dependencies-retries/Model the pipeline as task groups and give every task deliberate retries with backoff, a timeout and a pool — the skeleton's guarantees, verified the way the DAG reviewer verifies them before the DAG ships.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/agentic-capstone/https://emanisha.in/de-agent/tutorials/agentic-capstone/Run the whole series against one agent setup: skills that load on demand, tools scoped at the connection, least-privilege credentials, audited tool contracts, hooks that fail closed, and a human gate on anything irreversible. Prove each box with an attempted violation, not a promise.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/agentic-prod-gotchas/https://emanisha.in/de-agent/tutorials/agentic-prod-gotchas/An agent executes with your credentials, so its context and config are an attack surface. Read the MCP security model, treat fetched content as untrusted input, keep destructive statements deny-by-default, and scope every credential to the role and repositories the task needs.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/warehouse-production-gotchas/https://emanisha.in/de-agent/tutorials/warehouse-production-gotchas/The three ways a healthy warehouse model hurts production: a full-refresh that rebuilds a 4B-row table unannounced, a lookback window too short to catch late-arriving orders, and cost drift nobody reviews. Guard the destructive command, harden CI, and put the cost query on a weekly cadence.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/agentic-incident-rollback/https://emanisha.in/de-agent/tutorials/agentic-incident-rollback/An agent merged a change it should not have. Roll back first, use the scoping you configured to contain and reconstruct what happened — diffs, logs and approve-once records — walk the incident workflow, and close with a blameless postmortem whose action items name the missing gate.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/arrow-docs/https://emanisha.in/de-agent/references/arrow-docs/Official Arrow docs: the columnar in-memory format, its language bindings, IPC and Flight protocols, and zero-copy interchange between systems. Use it for exact memory-layout and type-system semantics behind every "columnar" claim in the registry.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/avro-docs/https://emanisha.in/de-agent/references/avro-docs/Avro's official documentation: schema declaration, the schema-registry evolution rules, encoding and the container format widely used for event/stream payloads and Kafka wire formats.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/parquet-format-docs/https://emanisha.in/de-agent/references/parquet-format-docs/The Parquet file format documentation: page structure, encodings, compression codecs, statistics and the predicate-pushdown behavior that columnar readers rely on. The ground truth for why columnar files scan less data and how row-group pruning works.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/aws-architecture-center/https://emanisha.in/de-agent/references/aws-architecture-center/AWS's official collection of architecture diagrams, reference architectures and best-practice guidance (Well-Architected pillars, serverless data lake patterns). Concrete vendor material for grounding sizing and tradeoff arguments in real deployed patterns.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/cloud-design-patterns/https://emanisha.in/de-agent/references/cloud-design-patterns/Microsoft's catalogue of cloud design patterns: retry, circuit breaker, compensating transaction, CQRS, event sourcing, bulkhead and more — each with problem/solution and considerations. Vendor-neutral in practice, and directly applicable to pipeline component design.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/sre-books/https://emanisha.in/de-agent/references/sre-books/Google's Site Reliability Engineering books, free online: SLIs/SLOs/error budgets, incident management, postmortem culture and capacity planning. The vocabulary source for reliability engineering that data platform teams borrow from when they define data SLAs and run incident drills.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/grafana-docs/https://emanisha.in/de-agent/references/grafana-docs/Grafana's official documentation: dashboards, panel queries, alerting (including the unified alerting model), and provisioning. The standard surface where pipeline freshness, volume and job-health dashboards live.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/opentelemetry-docs/https://emanisha.in/de-agent/references/opentelemetry-docs/OpenTelemetry's official documentation: traces, metrics and logs through a vendor-neutral API/SDK, the collector, and semantic conventions. Increasingly the default instrumentation layer for distributed pipelines and services.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/postgres-manual/https://emanisha.in/de-agent/references/postgres-manual/PostgreSQL's official documentation: the tutorial, SQL reference, indexes, transactions and MVCC chapters. The semantics of transaction isolation and index behavior here are the ground truth for correctness arguments in pipeline source systems.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/prometheus-docs/https://emanisha.in/de-agent/references/prometheus-docs/Official Prometheus docs: the pull-based metrics model, PromQL, exporters, service discovery and alerting rules. Data platforms lean on it for infrastructure- and job-level metrics that sit underneath data-quality signals like freshness and volume.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/python-docs/https://emanisha.in/de-agent/references/python-docs/The official Python 3 documentation: language reference, standard library (itertools, collections, dataclasses, json, csv), and the data-model chapter that explains iteration, mutability and hashing semantics every data script depends on.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/sqlbolt/https://emanisha.in/de-agent/references/sqlbolt/Free interactive SQL lessons: SELECT basics, filtering, joins, NULL handling and aggregate clauses, each runnable in the browser. A zero-setup first step before touching a real warehouse or local DuckDB.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/system-design-primer/https://emanisha.in/de-agent/references/system-design-primer/The widely-used open-source study guide for system design: scaling vocabulary (load balancers, caching, sharding), back-of-envelope estimation, and worked design exercises. Data platform architectures are a special case of the same throughput/consistency tradeoffs it teaches.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/twelve-factor/https://emanisha.in/de-agent/references/twelve-factor/The classic twelve principles for building services that run portably: config in the environment, backing services as attached resources, logs as event streams, disposability and dev/prod parity. Data pipelines inherit these ideas whenever they run as jobs on shared infrastructure.Fri, 23 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/author-first-data-contract/https://emanisha.in/de-agent/tutorials/author-first-data-contract/Draft an enforceable contract for one real dataset by walking the six parts — owner, schema, semantics, SLA, breaking-change policy, enforcement — starting from the data-contract template, and finish with the CI test that makes it a contract instead of documentation.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/cost-unit-economics-report/https://emanisha.in/de-agent/tutorials/cost-unit-economics-report/Turn cluster and warehouse billing into unit economics — cost per run, per 1M rows, per model build — so a regression shows up as a number, not a surprise invoice. The Cost Optimizer starts with attribution; this walkthrough ends with an alert on cost-per-run instead of total spend.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/athena-cut-bytes-scanned/https://emanisha.in/de-agent/tutorials/athena-cut-bytes-scanned/Walk one expensive Athena query through the Partition Advisor's levers in order: confirm the per-TB bill model, partition on the dominant filter column, replace MSCK REPAIR with partition projection, compact small files, and move to columnar compression. Every move is judged by a single number — bytes scanned.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/delta-optimize-decision/https://emanisha.in/de-agent/tutorials/delta-optimize-decision/Apply the Delta Optimization Advisor's decision rules to one table: read file stats for the OPTIMIZE trigger, pick Z-ORDER columns from actual query predicates, weigh liquid clustering for high-cardinality write-heavy tables, and prove the choice by comparing numFiles and bytes scanned before and after.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/dq-deduplicate-records/https://emanisha.in/de-agent/tutorials/dq-deduplicate-records/Row counts are inflated and revenue double-counted after a retry or a fan-out join. Work the duplicate-records runbook: find the first duplicate date, correlate it with a deploy, dedupe on the business key, and rewrite only the affected partitions.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/gate-contracts-in-ci/https://emanisha.in/de-agent/tutorials/gate-contracts-in-ci/Install the Schema Change Check as a pre-commit gate over staged schema files so a breaking change exits non-zero unless a migration plan or deprecation annotation rides along with it — the producer-CI placement that catches breakage before merge.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/sql-sqlfluff-core-rules/https://emanisha.in/de-agent/tutorials/sql-sqlfluff-core-rules/Install the SQLFluff core rule pack, point it at your warehouse dialect, and lint a fixture that fails on purpose: indentation, keyword capitalisation, SELECT star, and unused CTEs. You fix each finding while the passing fixture stays green, then wire the same lint run into CI.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/streaming-mental-model/https://emanisha.in/de-agent/tutorials/streaming-mental-model/Build the vocabulary before touching a cluster: events instead of rows, partitions instead of tables, at-least-once delivery instead of exactly-once, and the lambda/kappa decision for speed versus correctness. Every later lesson in this series leans on these four ideas.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/kafka-topics-via-mcp/https://emanisha.in/de-agent/tutorials/kafka-topics-via-mcp/Connect an agent to a Kafka cluster through the Kafka MCP server and work through its four tools: list_topics for inventory, describe_topic for a single topic, describe_consumer_group for lag, and consume_messages for sampling — stream debugging from an agent, with the sampling risks called out up front.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/cost-guardrails-setup/https://emanisha.in/de-agent/tutorials/cost-guardrails-setup/Instrument, attribute, then guard: the Cost Guardrails pattern in four moves — a unit metric per pipeline, a baseline of the last N runs, CI failing on regression, and hard caps. It ends with a build that fails on a 3.9x shuffle jump instead of a surprise invoice.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/agentic-agent-in-the-loop/https://emanisha.in/de-agent/tutorials/agentic-agent-in-the-loop/Coding agents can do real data-engineering work, but can also ship a schema change or drop a table. Walk the Agent-in-the-Loop Review pattern: separate read/recommend from write/execute, put a human gate before anything irreversible, and encode the policy as hooks.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/dq-four-dimensions/https://emanisha.in/de-agent/tutorials/dq-four-dimensions/Build the mental model before writing a single test: freshness, completeness, validity, consistency. See each dimension in a real registry artifact — the SLA part of a contract, the profiling workflow of the data-quality reviewer, the rule vocabulary of the quarantine starter — and leave able to name the dimension behind any "the data is bad" complaint.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/dq-respond-failing-expectation/https://emanisha.in/de-agent/tutorials/dq-respond-failing-expectation/A quality expectation failed in production and bad data may be reaching consumers. Follow the data-quality-failure runbook: read the failure precisely, quantify it, then make the one decision that matters — block, quarantine, or pass-with-warning.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/review-dag-idempotency/https://emanisha.in/de-agent/tutorials/review-dag-idempotency/Walk the Pipeline Reviewer's four-step pass over a DAG: apply the idempotency test to every task, check retry, timeout and concurrency settings, predict backfill behaviour, and propose missing integration tests — then close the loop with the Airflow DAG Reviewer's checklist so every task is safe to re-run before the DAG merges.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/glue-job-review-pass/https://emanisha.in/de-agent/tutorials/glue-job-review-pass/Take a Glue/Spark job through the AWS Glue Job Reviewer's three checks: size workers from the input files' partitions instead of habit, verify the job bookmark against the input path layout, and diff the Glue Data Catalog schema against the Parquet file footer. Each check targets a silent cost leak that ends in full reprocessing.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/dbt-warehouse-tests-starter/https://emanisha.in/de-agent/tutorials/dbt-warehouse-tests-starter/Clone the dbt-warehouse-tests starter, seed its deterministic customers and orders data, and build the staging → intermediate → marts pipeline with dbt build --target dev. You serve the docs, add the tests that protect the fct grain, and finish against the failure scenarios the project warns about.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/sql-review-full-pass/https://emanisha.in/de-agent/tutorials/sql-review-full-pass/Run the SQL Reviewer agent's full pass over a pull request: read the diff in context, state the grain of every result set, then check the correctness bugs the SQL Review skill targets — fan-out joins, NULL-unsafe NOT IN and = predicates — and emit findings with file:line and a rewritten clause you can verify against the evaluation fixture.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/scaffold-airflow-dag/https://emanisha.in/de-agent/tutorials/scaffold-airflow-dag/Generate a new pipeline from the Airflow DAG template and keep its guarantees — an idempotent skeleton with task groups, retries and backoff, SLA, pools, and a reschedule-mode sensor — then make catchup a deliberate choice and every write partition-keyed or a MERGE.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/s3-glue-athena-starter/https://emanisha.in/de-agent/tutorials/s3-glue-athena-starter/Deploy the serverless query-in-place starter end to end: Terraform provisions buckets, catalog, and a bounded Athena workgroup, Parquet lands in a Hive layout, the table is registered with partition projection, and pruning is proven with explain analyze. The sample data ships one deliberately oversized file to demonstrate compaction.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/spark-pyspark-template-start/https://emanisha.in/de-agent/tutorials/spark-pyspark-template-start/Scaffold a testable PySpark project with one command and learn where each piece lives: transforms in src/, 3-10 row fixtures in tests/data/, and a Makefile that runs lint plus tests offline. Finish by hardening the first transform with row-count and null assertions.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/batch-processing-mental-model/https://emanisha.in/de-agent/tutorials/batch-processing-mental-model/Batch processing in partition terms. Learn what a partition, a shuffle and a spill are, why "the job is slow" resolves into four named causes — skew, partition count, spill, over-scan — and why the write side carries one invariant: same date twice, same output.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/kafka-troubleshooter-workflow/https://emanisha.in/de-agent/tutorials/kafka-troubleshooter-workflow/Hand the Kafka Troubleshooter agent the consumer-group describe output, per-partition offsets, and consumer logs, then follow its four-step workflow from lag trend to the least destructive fix. The agent works read-only against the cluster, never resets offsets, and ships every fix with a risk rating.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/agentic-core-mcp-wiring/https://emanisha.in/de-agent/tutorials/agentic-core-mcp-wiring/Wire the three reference MCP servers a data repo needs: filesystem for the working directory, git for history and commits, and GitHub for issues and pull requests. Each connection is scoped — one directory, one repository, one fine-grained token.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/minimum-dbt-test-set/https://emanisha.in/de-agent/tutorials/minimum-dbt-test-set/Start from the model's grain, then rank dbt tests by the cost of the failure they prevent: uniqueness and not-null on the grain key, relationships on foreign keys, accepted values on enums, and freshness on sources. You add one custom singular test, wire freshness on a source, and finish by running the set in data-aware CI against a staging schema.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/spark-aggregations-windows/https://emanisha.in/de-agent/tutorials/spark-aggregations-windows/Aggregations and windows are shuffles in disguise. Treat the group-by key as a skew suspect, read p50 vs p99 and the spill bytes, then apply the matching fix — a salted two-stage aggregate, more partitions, or a partition predicate — and never repartition(1).Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/cost-attribute-4x-regression/https://emanisha.in/de-agent/tutorials/cost-attribute-4x-regression/One job's compute cost quadrupled overnight with no functional change. Run the 4x runbook: diff shuffle and input bytes run-over-run, find the guilty stage, reverse the regression — then wire hook-spark-cost-check so the next one fails CI before the invoice.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/cost-lens-plugin-walkthrough/https://emanisha.in/de-agent/tutorials/cost-lens-plugin-walkthrough/Install the Cost Lens plugin and use its three commands to attribute spend, compute unit economics, and guard cost-per-run. The plugin pairs the unit-economics skill with the Spark cost regression guard, and ships the hooks that enforce the guard in CI.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/agentic-mcp-contract-audit/https://emanisha.in/de-agent/tutorials/agentic-mcp-contract-audit/Before connecting an MCP server, audit its tools for blast radius. Read each tool schema, classify every tool as Read, Write, Destructive, Spend or Exfiltrate, and produce a least-privilege connection config that grants only what the task needs.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/batch-capstone-checklist/https://emanisha.in/de-agent/tutorials/batch-capstone-checklist/Bring the API → S3 → Spark → Parquet starter up end to end, then audit it: offline transform tests, idempotent partition-scoped writes, complete partitions, replay-safe extraction, and baseline metrics with a CI guard. One pipeline that exercises every lesson in the series.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/dq-capstone-checklist/https://emanisha.in/de-agent/tutorials/dq-capstone-checklist/Close the series by walking one SLA-bearing dataset through every gate the series taught — six-part contract, CI compatibility check, staged quality suite, quarantine split, quarantine-rate alerts — and finish with the monitoring signals and incident paths that keep the gates honest.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/lakehouse-guard-health/https://emanisha.in/de-agent/tutorials/lakehouse-guard-health/Use the Lakehouse Guard plugin's three commands on a Delta/Iceberg table: /table-health for a status read, /plan-compaction for a maintenance plan, and /expire-snapshots for a safe expiry plan. The guard exists to block compaction or snapshot expiry during peak ingestion windows, so plans cannot run in the wrong window.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/kafka-console-commands/https://emanisha.in/de-agent/tutorials/kafka-console-commands/Drive the Kafka Console plugin's three commands across a lag incident: /topic-info for inventory, /lag-triage for per-partition lag and rebalance detection, and /sample-topic for a read-only message sample with a PII redaction pass. Permissions stay at kafka-sasl-read throughout.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/spark-skew-diagnosis-walkthrough/https://emanisha.in/de-agent/tutorials/spark-skew-diagnosis-walkthrough/One Spark stage hangs while other executors idle. Open the stage view, read the p50-vs-p99 task durations, and follow the skew skill's decision order: AQE skew join first, salting second, broadcast only under a size budget — and never repartition(1).Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/small-files-diagnosis/https://emanisha.in/de-agent/tutorials/small-files-diagnosis/Follow the small-files runbook from symptom to prevention: confirm the tens-of-thousands-of-tiny-files signature, query the metadata tables for the average file size, compact to a 128-256 MB target, coarsen an over-partitioned layout, and schedule compaction outside the ingestion window. The fix follows the small-files compaction pattern — separate the write path from the layout path.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/gate-backfills-with-hook/https://emanisha.in/de-agent/tutorials/gate-backfills-with-hook/Wire the Backfill Safety Check as a pre-run gate that refuses any backfill lacking an explicit date range, a shadow or snapshotted target, and a window clear of the scheduled production run — and watch it block the bare backfill command that would otherwise corrupt the target.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/dq-pre-deploy-dq-gate/https://emanisha.in/de-agent/tutorials/dq-pre-deploy-dq-gate/Wire the pre-deployment DQ check so a pipeline is only promoted when its quality suite passes against staged output. You attach the pre-deploy trigger, run dbt/GE tests on the staging namespace, and prove a failed expectation leaves production untouched.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/incremental-watermark-batch/https://emanisha.in/de-agent/tutorials/incremental-watermark-batch/Full reloads stop scaling once a table grows past millions of rows. Process a bounded window instead: a watermark ends it, a lookback absorbs late files, and an idempotent write (partition overwrite or MERGE) makes every re-run safe — including re-processing a late window.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/streaming-idempotent-sink/https://emanisha.in/de-agent/tutorials/streaming-idempotent-sink/The core producer-side reliability skill for streams: you cannot stop replays, so the sink must converge. Deduplicate by a business key and event time within the watermark, MERGE on the key, and rank change events by (lsn, ts_ms) — never by arrival time.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/agentic-de-data-toolkit-start/https://emanisha.in/de-agent/tutorials/agentic-de-data-toolkit-start/Onboard an agent onto a data repo with the DE Data Toolkit plugin: the SQL/Spark review skills, the destructive-SQL guard, a secret scanner, and the file/git MCP servers already wired. Learn the three commands and which hooks run underneath.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/streaming-consumer-reliability/https://emanisha.in/de-agent/tutorials/streaming-consumer-reliability/Consumer reliability is where streams quietly break: a rebalance loop looks like slowness, consumers past the partition count buy nothing, and stop-the-world rebalances pause consumption. Read group state, fix the timeouts, and know what the next-generation protocol changes.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/plan-safe-backfill/https://emanisha.in/de-agent/tutorials/plan-safe-backfill/Use the Backfill Planner's four-step plan — correctness criteria, downstream impact from lineage, cost and contention, staged batches — then execute it the way the backfill runbook demands: shadow table, sample validation per batch, and never beside the daily production run.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/streaming-prod-gotchas/https://emanisha.in/de-agent/tutorials/streaming-prod-gotchas/The five causes of consumer lag on one card, the two destructive shortcuts that must never happen, and the capacity trap of scaling past the partition count. Ends with the alert that matters — lag slope, not absolute lag — and the sampling rules that keep PII out of agent contexts.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/spark-lens-event-log/https://emanisha.in/de-agent/tutorials/spark-lens-event-log/Install the Spark Lens plugin and work its three commands over a Spark event log: /spark-analyze to build a task histogram and flag skew, /spark-explain-plan to check the join strategy, and /spark-baseline to store the cost baseline the guard enforces.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/sql-query-plan-read/https://emanisha.in/de-agent/tutorials/sql-query-plan-read/Take the EXPLAIN output of a slow query and read it bottom-up, naming the expensive node before you touch anything. You compare estimated rows to actual rows to catch stale statistics, separate scan cost from compute time with the warehouse runbook, and finish by deciding between an index, a rewrite, or a stats refresh — one change, with a way to measure it.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/respond-broken-contract/https://emanisha.in/de-agent/tutorials/respond-broken-contract/Work the broken-contract runbook end to end: identify which clause broke, bound the consumer impact set from lineage before anything else, decide whether the producer reverts or the consumer adapts, and close with consumer-driven contract tests in the producer's pipeline.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/review-dbt-model-pr/https://emanisha.in/de-agent/tutorials/review-dbt-model-pr/Take a pull request that adds or changes a dbt model and walk the reviewer's pass: read the diff in context, state the model's grain and check it against the unique key test, choose table vs incremental vs view deliberately, and handle late-arriving data in the incremental filter. You finish with findings a reviewer can verify before the model merges.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/spark-dq-review-pass/https://emanisha.in/de-agent/tutorials/spark-dq-review-pass/Run the five-check pass over a PySpark transform before it merges: null propagation, join fan-out, silent coercion, non-determinism, and missing post-write assertions. Each finding lands as severity, file:line, rule and a suggested patch.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/review-schema-change-against-contract/https://emanisha.in/de-agent/tutorials/review-schema-change-against-contract/Run the Schema Reviewer's four-step pass over a producer's schema diff — classify each field change, resolve consumers from lineage, emit a dated migration plan — and use the schema-drift runbook to separate hard failures from silent nulls before anything merges.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/aws-architecture-review-pass/https://emanisha.in/de-agent/tutorials/aws-architecture-review-pass/Point the AWS Architecture Reviewer at an architecture description or IaC and follow its four-step workflow: map the services to a data flow, check IAM for least privilege, estimate dominant cost drivers, and identify single points of failure. The agent is read-only and never creates resources.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/dbt-workbench-commands/https://emanisha.in/de-agent/tutorials/dbt-workbench-commands/Install the dbt Workbench plugin and drive one model change through its three commands: /dbt-review-model for the model review, /dbt-write-tests for a proposed test set, and /dbt-impact before you merge. Along the way you apply the same grain and blast-radius checks the plugin's underlying skills use, with the full-refresh guard watching destructive rebuilds.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/iceberg-maintenance-routine/https://emanisha.in/de-agent/tutorials/iceberg-maintenance-routine/Work an Iceberg table through the maintenance skill's plan: read the files metadata table to size the problem, rewrite data files to a 128-256 MB target, rewrite manifests when the manifest list is huge, expire snapshots without breaking time travel, and remove orphan files on a schedule with a safety margin.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/dq-quarantine-publish-walkthrough/https://emanisha.in/de-agent/tutorials/dq-quarantine-publish-walkthrough/Build the pipeline-dq-quarantine starter the way the Quarantine then Publish pattern prescribes: a quality contract splits each batch into a published set and a quarantine with reasons. You finish with a test that proves no row is ever lost and an alert on the quarantine rate.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/streaming-cdc-lakehouse/https://emanisha.in/de-agent/tutorials/streaming-cdc-lakehouse/The intermediate pattern that replaces nightly extraction: read the write-ahead log through Debezium, stream through Kafka, dedupe by LSN, and MERGE into Delta/Iceberg. Covers tombstones, decimal handling, and what a dropped replication slot costs you.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/kafka-lag-triage-drill/https://emanisha.in/de-agent/tutorials/kafka-lag-triage-drill/Combine the lag-triage skill's three questions with the lagging-consumer runbook: read per-partition lag to separate capacity from key skew, check the group state history for a rebalance loop, find the one stuck offset that marks a poison message, then apply the matching fix — and never reset offsets.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/streaming-windows-watermarks/https://emanisha.in/de-agent/tutorials/streaming-windows-watermarks/Stream processing in practice: start from the kafka-streaming template, pick event-time over processing-time windows, set a watermark you can defend, and prove what happens to late data. Checkpointing and an idempotent MERGE sink make the job restartable without duplicating output.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/spark-incident-diagnosis/https://emanisha.in/de-agent/tutorials/spark-incident-diagnosis/The nightly job is crawling or dead under memory pressure or skew. Follow the incident path: collect the four pieces of evidence, read p50 vs p99 and the spill bytes, decide revert vs fix, escalate on golden-hour SLAs, and record the skew/spill/scan signature so the next run is faster to diagnose.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/spark-tuning-memory-partitions/https://emanisha.in/de-agent/tutorials/spark-tuning-memory-partitions/Run the tuning loop on a slow batch job: parse task-duration and shuffle metrics, classify the stage as skew, partition count, spill or scan, apply the single highest-impact change, and verify it against a stored baseline the CI guard enforces.Wed, 21 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/tutorials/sql-fanout-review/https://emanisha.in/de-agent/tutorials/sql-fanout-review/Follow the SQL Reviewer agent's four-step pass over a real query: read the diff and its context, establish the grain of every result set, apply the SQL Review skill's five rules while stopping at the first fan-out, and emit findings with file positions plus a rewritten clause you can verify against the twelve-case evaluation fixture.Fri, 16 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/mcp/datahub/https://emanisha.in/de-agent/mcp/datahub/Exposes a DataHub catalog: search entities, read lineage and schema, and inspect metadata, so an agent can answer "what is this table and what depends on it?".Sat, 10 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/mcp/dbt/https://emanisha.in/de-agent/mcp/dbt/dbt Labs server that exposes project discovery, the semantic layer, and the dbt CLI (`run`, `test`, `compile`) as tools, so an agent can build and inspect models.Sat, 10 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/mcp/aws/https://emanisha.in/de-agent/mcp/aws/A suite of AWS Labs servers — including documentation search, CDK/Terraform assistance, and service-specific servers — for agents working on AWS.Sat, 10 Oct 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/sql-review/https://emanisha.in/de-agent/skills/sql-review/Reviews a SQL statement for fan-out joins, NULL-unsafe predicates, implicit casts, non-deterministic ordering, and functions that silently change grain.Wed, 30 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/spark-data-quality-review/https://emanisha.in/de-agent/skills/spark-data-quality-review/A review skill an agent runs against a Spark/PySpark transform. It checks null propagation, join fan-out, silent type coercion, non-deterministic row counts and missing post-write assertions, then reports findings as a diff-level list.Wed, 30 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/mcp/fetch/https://emanisha.in/de-agent/mcp/fetch/Retrieves a URL and converts it to markdown for the model. Commonly used to read documentation during a task.Wed, 30 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/mcp/filesystem/https://emanisha.in/de-agent/mcp/filesystem/Reference server that exposes a sandboxed set of directories as readable and (optionally) writable tools. Useful for pipelines that operate on local files.Wed, 30 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/mcp/git/https://emanisha.in/de-agent/mcp/git/Exposes local Git operations — status, diff, log, commit, branch — as tools, so an agent can inspect history and prepare changes without shelling out blindly.Wed, 30 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/mcp/postgres/https://emanisha.in/de-agent/mcp/postgres/Connects an agent to a Postgres database with a read-only query tool and schema introspection. The canonical example of a "database over MCP" server.Wed, 30 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/plugins/de-data-toolkit/https://emanisha.in/de-agent/plugins/de-data-toolkit/The opinionated starter plugin: SQL/Spark review skills, the destructive-SQL guard, a secret scanner and the file/git MCP servers, wired for data engineers.Wed, 30 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/agents/sql-reviewer/https://emanisha.in/de-agent/agents/sql-reviewer/Review SQL before it merges, catching correctness bugs and fan-out joins.Wed, 30 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/templates/docker-local-stack/https://emanisha.in/de-agent/templates/docker-local-stack/A docker-compose stack (Postgres, Kafka, MinIO, Spark, and a DuckDB CLI) for developing and testing pipelines entirely offline.Wed, 30 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/duckdb-docs/https://emanisha.in/de-agent/references/duckdb-docs/DuckDB's documentation — the embedded analytics SQL engine, Parquet and CSV readers, in-process query execution, and extensions (ICU, JSON, Parquet, Excel). The fastest way to make a pipeline testable locally; covers everything our SQL review and testing skills reference, plus the DuckDB docs themselves.Wed, 30 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/mcp-specification/https://emanisha.in/de-agent/references/mcp-specification/The Model Context Protocol Specification — definitive reference for transports (stdio, SSE, HTTP), tool definitions, resource templates, prompts, and the security model. The security section is required reading before connecting any MCP server to real data; the transport section explains stdio vs HTTP tradeoffs. The canonical reference for all MCP server implementations on this platform.Wed, 30 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/mcp-tool-contract-auditor/https://emanisha.in/de-agent/skills/mcp-tool-contract-auditor/Inspects an MCP server's tool list and schemas, flags tools that can write, delete, spend money or exfiltrate data, and produces a least-privilege connection recommendation.Tue, 29 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/mcp/github/https://emanisha.in/de-agent/mcp/github/First-party GitHub server exposing repositories, issues, pull requests and code search, plus optional write operations.Tue, 29 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/hooks/destructive-sql-guard/https://emanisha.in/de-agent/hooks/destructive-sql-guard/Intercept DROP / TRUNCATE / DELETE-without-WHERE before an agent executes it.Tue, 29 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/runbooks/pipeline-failed/https://emanisha.in/de-agent/runbooks/pipeline-failed/A task or DAG run has failed and downstream work is blocked — triage, classify and recover.Tue, 29 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/patterns/agent-in-the-loop-review/https://emanisha.in/de-agent/patterns/agent-in-the-loop-review/Coding agents can do real data-engineering work, but can also ship a schema change or drop a table.Tue, 29 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/airflow-docs/https://emanisha.in/de-agent/references/airflow-docs/Airflow's documentation — DAG authoring, scheduling semantics and timetables, backfills and catchup, task state and retries, dynamic task mapping, and best practices. The scheduling and best-practices pages answer the idempotency and backfill questions our orchestration skills only sketch; read them before designing any recurring pipeline.Tue, 29 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/mcp-registry/https://emanisha.in/de-agent/references/mcp-registry/The official MCP Registry — community-driven registry of MCP servers with a metadata schema for discovery. The canonical place to discover MCP servers — treat entries as *discovery*, never as a trust signal. The metadata schema defines server capabilities, transport, and authentication; essential for building MCP-aware tooling.Tue, 29 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/dbt-test-writer/https://emanisha.in/de-agent/skills/dbt-test-writer/Proposes dbt tests ranked by blast radius: uniqueness/not-null on the grain key, accepted values on enums, relationships on foreign keys, and freshness on sources.Mon, 28 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/dbt-model-review/https://emanisha.in/de-agent/skills/dbt-model-review/Checks a dbt model's grain statement, ref/dependency hygiene, incremental strategy and late-arriving-data handling, then proposes a test set.Mon, 28 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/mcp/duckdb/https://emanisha.in/de-agent/mcp/duckdb/Queries DuckDB databases and MotherDuck, including reading Parquet/CSV directly from object storage. Excellent for local, offline analysis with an agent.Mon, 28 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/plugins/dbt-workbench/https://emanisha.in/de-agent/plugins/dbt-workbench/Model review, test generation, incremental-safety checks and a full-refresh guard, plus the dbt MCP server for running commands from the agent.Mon, 28 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/hooks/schema-change-check/https://emanisha.in/de-agent/hooks/schema-change-check/Block a commit that introduces a breaking schema change without a migration plan.Mon, 28 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/agents/spark-optimizer/https://emanisha.in/de-agent/agents/spark-optimizer/Diagnose and fix slow or expensive Spark jobs from their metrics.Mon, 28 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/templates/pyspark-project/https://emanisha.in/de-agent/templates/pyspark-project/A testable PySpark project: src/transforms, tests with a local Spark session, a Dockerfile, and a Makefile that runs lint + tests offline.Mon, 28 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/projects/api-s3-spark-parquet/https://emanisha.in/de-agent/projects/api-s3-spark-parquet/Land a paginated REST API into S3 raw, transform with PySpark into partitioned Parquet, and make the whole thing idempotent and testable offline.Mon, 28 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/runbooks/spark-job-slow/https://emanisha.in/de-agent/runbooks/spark-job-slow/A batch Spark job's runtime has jumped — find whether it is skew, partition count, spill or over-scan.Mon, 28 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/patterns/medallion-zones/https://emanisha.in/de-agent/patterns/medallion-zones/Object storage gives you a place to put files but no structure for trust or ownership.Mon, 28 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/agent-skills/https://emanisha.in/de-agent/references/agent-skills/The Agent Skills format specification — a SKILL.md file with front-matter (name, description, version), plus references/, scripts/, and examples/ directories that an agent loads on demand. The format this platform's Skill object is modelled on — portable across compatible agents. Covers the SKILL.md schema, the directory layout convention, and the agent loading protocol.Mon, 28 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/spark-docs/https://emanisha.in/de-agent/references/spark-docs/The authoritative reference for Spark SQL, the DataFrame API, tuning and the configuration catalogue, versioned per Spark release. Use it for exact semantics and config keys — when a job skews or slows, check the config reference and the tuning guide before changing code, alongside our skew-diagnosis and cost skills.Mon, 28 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/pipeline-test-author/https://emanisha.in/de-agent/skills/pipeline-test-author/Builds a local DuckDB/pyarrow harness with tiny fixtures so pipeline logic is testable in CI without a cluster, and asserts on invariants, not snapshots.Sun, 27 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/mcp/snowflake/https://emanisha.in/de-agent/mcp/snowflake/Snowflake Labs server exposing object management and SQL execution against a Snowflake account, including Cortex Search and Analyst services.Sun, 27 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/plugins/contract-kit/https://emanisha.in/de-agent/plugins/contract-kit/Authors data contracts, enforces them in CI, and classifies schema changes as compatible/risky/breaking for producers and consumers.Sun, 27 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/hooks/pre-deployment-dq-check/https://emanisha.in/de-agent/hooks/pre-deployment-dq-check/Run the quality suite against a staged run before promoting to production.Sun, 27 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/templates/dbt-project/https://emanisha.in/de-agent/templates/dbt-project/A layered dbt project (staging/intermediate/marts) with tests, docs, a CI workflow that builds against a staging schema, and a source freshness config.Sun, 27 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/projects/dbt-warehouse-tests/https://emanisha.in/de-agent/projects/dbt-warehouse-tests/Transform raw landed data into a tested, documented star schema with dbt, and gate every change on data-aware CI.Sun, 27 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/runbooks/data-quality-failure/https://emanisha.in/de-agent/runbooks/data-quality-failure/A quality expectation failed in production and bad data may be reaching consumers.Sun, 27 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/runbooks/kafka-consumer-lagging/https://emanisha.in/de-agent/runbooks/kafka-consumer-lagging/A consumer group is falling behind the topic and downstream tables are going stale.Sun, 27 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/patterns/quarantine-publish/https://emanisha.in/de-agent/patterns/quarantine-publish/One bad batch of rows can poison a whole table and every downstream metric.Sun, 27 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/kafka-docs/https://emanisha.in/de-agent/references/kafka-docs/Kafka's official documentation — architecture and design, delivery semantics, the protocol, consumer groups and offsets, operations, and security. The design section explains why ordering and partitions behave the way they do; the operations section companions our lag-triage skill and consumer-lag runbook when a group falls behind.Sun, 27 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/dbt-docs/https://emanisha.in/de-agent/references/dbt-docs/dbt's documentation — the analytics-engineering workflow, project structure with models and sources, materialisations and incremental-model patterns, tests (unique, not-null, relationships, accepted values) and the docs-as-data approach. The materialisation and incremental-model pages are the reference behind our dbt review and testing skills; pair with our warehouse-testing guide for the practical path.Sun, 27 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/schema-change-contract-review/https://emanisha.in/de-agent/skills/schema-change-contract-review/Diffs an old and new schema, classifies the change (compatible / risky / breaking) against consumers, and produces the migration and comms plan.Sat, 26 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/plugins/spark-lens/https://emanisha.in/de-agent/plugins/spark-lens/Parses Spark event logs into a task histogram, flags skew and shuffle amplification, and compares against a stored cost baseline.Sat, 26 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/hooks/secret-scan-commit/https://emanisha.in/de-agent/hooks/secret-scan-commit/Stop credentials from being committed into a pipeline repo.Sat, 26 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/agents/data-quality-reviewer/https://emanisha.in/de-agent/agents/data-quality-reviewer/Audit a dataset or pipeline for the quality failures that reach consumers.Sat, 26 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/projects/pipeline-dq-quarantine/https://emanisha.in/de-agent/projects/pipeline-dq-quarantine/Put a contract in front of a table: validate each batch, publish the clean rows, quarantine the rest with reasons, and alert on the quarantine rate.Sat, 26 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/runbooks/schema-drift/https://emanisha.in/de-agent/runbooks/schema-drift/The upstream schema changed without notice and a cast broke or a column silently nulled.Sat, 26 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/patterns/dbt-tests-docs/https://emanisha.in/de-agent/patterns/dbt-tests-docs/Transformation logic lives in untested, undocumented SQL scattered across scripts.Sat, 26 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/patterns/batch-incremental-watermark/https://emanisha.in/de-agent/patterns/batch-incremental-watermark/Full reloads are too slow and too expensive once a table grows past millions of rows.Sat, 26 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/dbt-best-practices/https://emanisha.in/de-agent/references/dbt-best-practices/dbt Labs' official best practices guide — project structure, staging/marts layering, naming conventions, testing conventions (unique, not_null, relationships, accepted_values), model contracts, and incremental model patterns. A widely-adopted convention set; agreeing on it early avoids enormous friction later. The testing conventions section directly backs our dbt test writer skill; the materialization guide supports our warehouse testing patterns.Sat, 26 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/delta-lake-docs/https://emanisha.in/de-agent/references/delta-lake-docs/Delta Lake's official docs — ACID transactions on object storage, time travel, OPTIMIZE and Z-ORDER clustering, Delta Live Tables, and Unity Catalog integration. The definitive source for Delta features and the maintenance operations our lakehouse skills reference; compare with our Delta vs Iceberg decision guide before committing to a format.Sat, 26 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/incident-postmortem-writer/https://emanisha.in/de-agent/skills/incident-postmortem-writer/Structures incident notes into a blameless postmortem: timeline, impact, detection gap, contributing factors, and action items with owners and dates.Fri, 25 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/plugins/runbook-runner/https://emanisha.in/de-agent/plugins/runbook-runner/Loads the production playbooks into the agent, matches an alert to a runbook, and walks the operator through diagnosis and mitigation step by step.Fri, 25 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/hooks/prod-write-confirm/https://emanisha.in/de-agent/hooks/prod-write-confirm/Force human confirmation for any write against a production data target.Fri, 25 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/agents/incident-assistant/https://emanisha.in/de-agent/agents/incident-assistant/Guide an on-call engineer through a data incident and produce the postmortem.Fri, 25 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/projects/postgres-cdc-kafka-lakehouse/https://emanisha.in/de-agent/projects/postgres-cdc-kafka-lakehouse/Capture changes from Postgres with Debezium, stream through Kafka, and merge into a lakehouse table with correct ordering, deletes and replay safety.Fri, 25 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/runbooks/backfill-required/https://emanisha.in/de-agent/runbooks/backfill-required/Historical output is wrong or missing and must be recomputed safely, without breaking production.Fri, 25 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/runbooks/spark-job-suddenly-costs-4x/https://emanisha.in/de-agent/runbooks/spark-job-suddenly-costs-4x/One job's compute cost quadrupled without an obvious functional change — attribute it and reverse the regression.Fri, 25 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/patterns/postgres-cdc-lakehouse/https://emanisha.in/de-agent/patterns/postgres-cdc-lakehouse/Batch extraction from an OLTP database loads it and still misses deletes and mid-day changes.Fri, 25 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/iceberg-docs/https://emanisha.in/de-agent/references/iceberg-docs/Iceberg's specification and documentation — the table format, snapshots and manifests, hidden partitioning, schema and partition evolution, and maintenance procedures like compaction and snapshot expiry. The table format is a specification, so the spec is the ground truth when engines disagree; essential reading for multi-engine shops where Spark, Trino, and warehouses share tables.Fri, 25 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/airflow-dag-reviewer/https://emanisha.in/de-agent/skills/airflow-dag-reviewer/Checks a DAG for non-idempotent tasks, unsafe retries, missing catchup policy, ambiguous catchup/backfill semantics and unbounded concurrency.Thu, 24 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/hooks/spark-cost-check/https://emanisha.in/de-agent/hooks/spark-cost-check/Fail a build when a Spark job's shuffle/spill/scan metrics regress past a threshold.Thu, 24 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/agents/schema-reviewer/https://emanisha.in/de-agent/agents/schema-reviewer/Classify schema changes and prevent breaking changes from shipping.Thu, 24 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/templates/airflow-dag/https://emanisha.in/de-agent/templates/airflow-dag/An idempotent DAG skeleton with task groups, retries and backoff, SLA, pools, and a sensor using reschedule mode.Thu, 24 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/runbooks/cdc-stopped/https://emanisha.in/de-agent/runbooks/cdc-stopped/Change events stopped flowing from a source and the target table is silently diverging.Thu, 24 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/patterns/idempotent-partition-overwrite/https://emanisha.in/de-agent/patterns/idempotent-partition-overwrite/Retries and backfills duplicate or corrupt data because writes are appends.Thu, 24 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/patterns/backfill-safety/https://emanisha.in/de-agent/patterns/backfill-safety/Backfills corrupt production or take down the daily run.Thu, 24 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/debezium-docs/https://emanisha.in/de-agent/references/debezium-docs/Debezium's documentation — source and sink connectors, the change-event envelope format, initial snapshots, schema history handling, and the operational pitfalls of running connectors in production. The connector matrix and the events section are what you need to design a correct CDC merge; pair with our Postgres CDC lakehouse project for the full path.Thu, 24 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/snowflake-docs/https://emanisha.in/de-agent/references/snowflake-docs/Snowflake's official documentation — comprehensive coverage of micro-partitions, clustering keys, virtual warehouses, streams and tasks, time travel, and zero-copy cloning. The performance and cost sections are the definitive reference for clustering-key design, warehouse sizing, and query optimization. The Snowflake Cortex and ML functions sections document built-in ML capabilities. The security chapter covers RBAC, network policies, and Tri-Secret Secure. Essential reading for anyone operating Snowflake at scale; pairs with our cost-engineering and warehouse-clustering skills.Thu, 24 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/plugins/warehouse-explorer/https://emanisha.in/de-agent/plugins/warehouse-explorer/Schema browsing, query-plan explanation, clustering advice and a read-only connection profile for Snowflake/BigQuery/Redshift.Wed, 23 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/agents/pipeline-reviewer/https://emanisha.in/de-agent/agents/pipeline-reviewer/Review a DAG or pipeline for idempotency, retries and backfill safety.Wed, 23 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/templates/data-contract-template/https://emanisha.in/de-agent/templates/data-contract-template/A contract skeleton (ownership, schema, semantics, SLA, compatibility policy) with the CI check that enforces it and a deprecation checklist.Wed, 23 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/projects/kafka-spark-delta/https://emanisha.in/de-agent/projects/kafka-spark-delta/A low-latency stream into Delta Lake with watermarking, windowed aggregation and idempotent upserts that survive replays.Wed, 23 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/runbooks/duplicate-records/https://emanisha.in/de-agent/runbooks/duplicate-records/Row counts are inflated and metrics double-counted after a retry, a fan-out join or a backfill.Wed, 23 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/patterns/streaming-dedup-upsert/https://emanisha.in/de-agent/patterns/streaming-dedup-upsert/Streaming sinks either duplicate on replay or lose rows on failure.Wed, 23 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/sql-query-plan-explainer/https://emanisha.in/de-agent/skills/sql-query-plan-explainer/Reads EXPLAIN / EXPLAIN ANALYZE output and explains scans, joins, spills and estimated-vs-actual row skew, then proposes the single highest-impact change.Tue, 22 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/hooks/backfill-safety-check/https://emanisha.in/de-agent/hooks/backfill-safety-check/Refuse to start a backfill that lacks a rollback path or overlaps the daily run.Tue, 22 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/projects/s3-glue-athena/https://emanisha.in/de-agent/projects/s3-glue-athena/A serverless, query-in-place pattern: land Parquet in S3, catalogue it with Glue, and query with Athena while keeping scanned bytes (and cost) low.Tue, 22 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/runbooks/small-files-explosion/https://emanisha.in/de-agent/runbooks/small-files-explosion/A table has fragmented into thousands of tiny files and query planning now dominates runtime.Tue, 22 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/patterns/small-files-compaction/https://emanisha.in/de-agent/patterns/small-files-compaction/High-frequency writes shatter a table into thousands of tiny files and queries crawl.Tue, 22 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/patterns/contracts-quality-gate/https://emanisha.in/de-agent/patterns/contracts-quality-gate/Producers change schemas and semantics with no notice; consumers break in production.Tue, 22 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/great-expectations-docs/https://emanisha.in/de-agent/references/great-expectations-docs/Great Expectations documentation — expectations (expectations, suites, checkpoints) as a YAML DSL for data quality, plus the data-doc approach that connects quality rules to documentation. The expectation catalogue is a practical vocabulary for expressing rules with precision; pairs with our Soda Core reference and data-quality starter guide.Tue, 22 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/aws-glue-job-reviewer/https://emanisha.in/de-agent/skills/aws-glue-job-reviewer/Reviews Glue/Spark jobs for worker sizing, job bookmarks, partition pruning, and the catalogue/schema pitfalls that cause silent full reprocessing.Mon, 21 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/plugins/kafka-console/https://emanisha.in/de-agent/plugins/kafka-console/Topic inventory, per-partition lag triage, rebalance detection and a read-only message sampler with a PII redaction pass.Mon, 21 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/hooks/dbt-full-refresh-guard/https://emanisha.in/de-agent/hooks/dbt-full-refresh-guard/Require confirmation before a full-refresh of a large incremental model.Mon, 21 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/agents/aws-architecture-reviewer/https://emanisha.in/de-agent/agents/aws-architecture-reviewer/Review an AWS data architecture for cost, IAM and reliability risks.Mon, 21 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/templates/kafka-streaming/https://emanisha.in/de-agent/templates/kafka-streaming/A structured-streaming job with checkpointing, watermarking, a dedupe stage and an idempotent MERGE sink to Delta/Iceberg.Mon, 21 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/runbooks/data-contract-broken/https://emanisha.in/de-agent/runbooks/data-contract-broken/A producer violated its contract and a consumer broke in production — bound the impact before fixing.Mon, 21 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/bigquery-docs/https://emanisha.in/de-agent/references/bigquery-docs/BigQuery's official documentation — comprehensive coverage of partitioning, clustering, slot management, and the query optimizer's behavior. Covers BigQuery ML, BI Engine, materialized views, and authorized views. The partitioning/clustering pages explain the exact levers for cost control; the query optimizer section explains slot utilization and query planning. Essential for our Athena/BigQuery cost comparison guide and BigQuery cost-optimization patterns.Mon, 21 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/kafka-consumer-lag-triage/https://emanisha.in/de-agent/skills/kafka-consumer-lag-triage/Triages consumer lag from the consumer-group offsets: distinguishing a slow consumer, a partitioned hot key, a rebalance storm, and a poison-pill retry loop.Sun, 20 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/mcp/bigquery/https://emanisha.in/de-agent/mcp/bigquery/Community server for BigQuery that lists datasets and runs SQL against a project, with support for the BigQuery SQL dialect.Sun, 20 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/plugins/lineage-graph/https://emanisha.in/de-agent/plugins/lineage-graph/Column-level impact analysis over a lineage store, with a change-impact command that lists the downstream consumers a schema change would break.Sun, 20 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/hooks/pii-leak-guard/https://emanisha.in/de-agent/hooks/pii-leak-guard/Prevent raw PII from entering model context or being written to an unprotected path.Sun, 20 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/agents/kafka-troubleshooter/https://emanisha.in/de-agent/agents/kafka-troubleshooter/Triage consumer lag, rebalance loops and poison messages in minutes.Sun, 20 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/templates/terraform-aws-data-platform/https://emanisha.in/de-agent/templates/terraform-aws-data-platform/S3 buckets with lifecycle + encryption, Glue catalogue, IAM roles scoped to least privilege, and Athena workgroups with query limits.Sun, 20 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/runbooks/bad-deployment/https://emanisha.in/de-agent/runbooks/bad-deployment/A deploy broke production; correlate the signal with the change and roll back first.Sun, 20 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/patterns/dimensional-model/https://emanisha.in/de-agent/patterns/dimensional-model/Analysts re-derive the same joins and disagree on metric definitions.Sun, 20 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/parquet-docs/https://emanisha.in/de-agent/references/parquet-docs/Apache Parquet documentation and specification — columnar encoding, row groups, statistics, predicate pushdown, and how to structure files for efficient reads. Understanding statistics and row-group sizing is the practical step that turns "use Parquet" into "and lay out the file like this"; pairs with small-files and lambda-vs-kappa guides.Sun, 20 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/backfill-planner/https://emanisha.in/de-agent/skills/backfill-planner/Produces a backfill plan: date ranges, blast radius, resource contention, idempotency check, validation gates and rollback.Sat, 19 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/plugins/lakehouse-guard/https://emanisha.in/de-agent/plugins/lakehouse-guard/Delta/Iceberg table maintenance advisor plus a guard that blocks compaction or snapshot expiry during peak ingestion windows.Sat, 19 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/agents/backfill-planner-agent/https://emanisha.in/de-agent/agents/backfill-planner-agent/Produce a safe, staged backfill plan with validation and rollback.Sat, 19 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/runbooks/late-files/https://emanisha.in/de-agent/runbooks/late-files/Partner files arrive after the job has already run, so historical numbers keep changing.Sat, 19 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/patterns/cost-guardrails/https://emanisha.in/de-agent/patterns/cost-guardrails/A working pipeline silently becomes expensive and nobody notices until the invoice.Sat, 19 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/databricks-docs/https://emanisha.in/de-agent/references/databricks-docs/Databricks Documentation — comprehensive coverage of Delta Live Tables, Unity Catalog, jobs and clusters, liquid clustering, Photon engine, and Unity Catalog governance. The Liquid Clustering and Unity Catalog pages are platform-specific and only documented here; the Delta Live Tables section is the definitive reference for building reliable pipelines. Essential for our lakehouse skills and Databricks-specific patterns.Sat, 19 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/spark-skew-diagnosis/https://emanisha.in/de-agent/skills/spark-skew-diagnosis/Walks an agent through reading the Spark UI/event log to distinguish key skew, partition skew, and shuffle-write amplification, then selects between salting, AQE skew join, broadcast, and repartition strategies.Fri, 18 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/hooks/freshness-pre-merge/https://emanisha.in/de-agent/hooks/freshness-pre-merge/Warn when a change would silently widen the freshness SLA of a source.Fri, 18 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/agents/databricks-reviewer/https://emanisha.in/de-agent/agents/databricks-reviewer/Review notebooks, jobs and Delta tables for the Databricks-specific pitfalls.Fri, 18 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/runbooks/warehouse-query-slow/https://emanisha.in/de-agent/runbooks/warehouse-query-slow/A query that used to be fast became slow — separate scan inefficiency from compute contention.Fri, 18 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/datahub-docs/https://emanisha.in/de-agent/references/datahub-docs/DataHub's documentation — the open metadata model for ownership and lineage, ingestion framework, search, and governance integration. Explains how to represent ownership and lineage consistently across tools; the key vocabulary resource for any catalog-related pipeline design.Fri, 18 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/warehouse-clustering-advisor/https://emanisha.in/de-agent/skills/warehouse-clustering-advisor/Advises on Snowflake/BigQuery/Redshift tuning: clustering keys, warehouse sizing, result caching, and the difference between a scan problem and a compute problem.Thu, 17 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/plugins/cost-lens/https://emanisha.in/de-agent/plugins/cost-lens/Attributes warehouse/cluster spend to pipelines, computes unit economics, and alerts on cost-per-run regressions rather than on total spend.Thu, 17 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/patterns/lambda-vs-kappa/https://emanisha.in/de-agent/patterns/lambda-vs-kappa/Streaming is fast but hard; batch is simple but slow; you need both speed and correctness.Thu, 17 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/aws-glue-docs/https://emanisha.in/de-agent/references/aws-glue-docs/AWS Glue Documentation — complete reference for Glue jobs, crawlers, the Data Catalog, job bookmarks, DPU configuration, and Ray integration. The job bookmarks and catalog behavior sections are the source of the most common Glue surprises; the Spark UI integration pages explain how to debug slow jobs. Essential for our Glue review skills and CDC pipeline patterns.Thu, 17 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/iceberg-table-maintenance/https://emanisha.in/de-agent/skills/iceberg-table-maintenance/Diagnoses small-file and manifest bloat and generates the compaction, snapshot expiry and orphan-file cleanup plan for an Iceberg table.Wed, 16 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/agents/cost-optimizer/https://emanisha.in/de-agent/agents/cost-optimizer/Reduce the unit cost of pipelines without breaking them.Wed, 16 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/runbooks/missing-partition/https://emanisha.in/de-agent/runbooks/missing-partition/An expected date partition is absent, so a downstream report silently excludes a day.Wed, 16 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/openlineage-docs/https://emanisha.in/de-agent/references/openlineage-docs/OpenLineage documentation — the open specification for lineage events, plus integrations for Airflow, Spark and dbt. Not a tool or a service; it is a standard for emitting lineage events, which is what makes cross-tool lineage possible. Read it before wiring any line-tracking pipeline.Wed, 16 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/cdc-merge-designer/https://emanisha.in/de-agent/skills/cdc-merge-designer/Produces a MERGE (or Delta/Iceberg merge) that applies inserts, updates and deletes from a CDC stream correctly, including out-of-order events and tombstones.Tue, 15 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/mcp/kafka/https://emanisha.in/de-agent/mcp/kafka/Community server for Apache Kafka that lists topics, reads consumer-group lag, and samples messages, for stream debugging from an agent.Tue, 15 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/delta-optimization-advisor/https://emanisha.in/de-agent/skills/delta-optimization-advisor/Reads Delta table history and file statistics to decide between OPTIMIZE, Z-ORDER, liquid clustering and partitioning changes.Mon, 14 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/spark-cost-regression-guard/https://emanisha.in/de-agent/skills/spark-cost-regression-guard/Compares a job's shuffle/spill/scan metrics against a stored baseline and fails the change if cost-per-run regresses beyond a threshold, with the diff explained.Mon, 14 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/patterns/data-vault/https://emanisha.in/de-agent/patterns/data-vault/Integrating many changing sources into a warehouse without constant rework.Mon, 14 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/soda-core/https://emanisha.in/de-agent/references/soda-core/Soda Core — open-source checks-as-code for data quality with a YAML DSL for column and row-level checks, metrics, and thresholds. The SQL-metric approach is easier to reason about than Great Expectations' expectation suites; the programmatic API enables CI/CD integration. Lightweight alternative to Great Expectations; pairs with our data-quality starter guide and Great Expectations reference.Mon, 14 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/data-contract-author/https://emanisha.in/de-agent/skills/data-contract-author/Drafts a data contract (schema, semantics, freshness SLA, ownership, breaking- change policy) and the tests that enforce it in CI.Sun, 13 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/athena-partition-advisor/https://emanisha.in/de-agent/skills/athena-partition-advisor/Recommends partition projection, bucketing or file-layout changes to reduce Athena bytes-scanned, and writes the projection DDL.Sat, 12 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/lineage-explainer/https://emanisha.in/de-agent/skills/lineage-explainer/Traces column-level lineage from an asset to its upstream sources and downstream consumers, and reports the impact set for a proposed change.Sat, 12 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/agents/data-catalog-assistant/https://emanisha.in/de-agent/agents/data-catalog-assistant/Answer "what is this data, who owns it, and is it safe to use?".Sat, 12 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/data-mesh/https://emanisha.in/de-agent/references/data-mesh/Organizational principles for analytical data at scale — domain ownership, data as a product, self-serve data platform, and federated computational governance. Zhamak Dehghani's Data Mesh provides the organizational counterpoint to a central platform team; the vocabulary and principles are essential for governance discussions and platform strategy. The four principles (domain ownership, data as product, self-serve platform, federated governance) map directly to our governance and platform patterns.Sat, 12 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/designing-data-intensive-applications/https://emanisha.in/de-agent/references/designing-data-intensive-applications/The canonical book on the principles behind data systems: reliability, scalability, replication, partitioning, consistency, and batch/stream processing. Martin Kleppmann's DDIA is the single highest-value read for system-design interviews and for understanding why data tools are shaped as they are. Every chapter maps directly to patterns in this registry — replication to CDC, partitioning to lakehouse, consistency to transaction semantics. The definitive textbook for data engineers.Sat, 12 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/references/fundamentals-of-data-engineering/https://emanisha.in/de-agent/references/fundamentals-of-data-engineering/A lifecycle view of data engineering — generation, storage, ingestion, transformation, and serving — with the "undercurrents" of security, governance, and cost. Joe Reis and Matt Housley provide a modern, practitioner-oriented map of the field. The best overview for someone moving into data engineering; pairs perfectly with this platform's roadmap. Covers the full lifecycle from ingestion through serving, with practical patterns for each stage.Sat, 12 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/cost-unit-economics-reporter/https://emanisha.in/de-agent/skills/cost-unit-economics-reporter/Turns cluster/warehouse billing into unit economics (cost per 1M rows, per run, per model) so a 4x regression shows up as a number, not a surprise invoice.Fri, 11 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/skills/pii-classifier/https://emanisha.in/de-agent/skills/pii-classifier/Scans column names and sample values for PII categories, then proposes classification tags and a masking/pseudonymisation policy.Thu, 10 Sep 2026 00:00:00 GMThttps://emanisha.in/de-agent/resources/sqlfluff-core-rules/https://emanisha.in/de-agent/resources/sqlfluff-core-rules/A curated SQLFluff configuration with rules optimized for modern data engineering workflows. Includes dialect-specific rules for dbt (Jinja), Spark SQL, Snowflake, BigQuery, and Redshift. Enforces naming conventions, formatting standards, performance anti-patterns, and security rules.Sat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/resources/data-classification-governance/https://emanisha.in/de-agent/resources/data-classification-governance/A complete governance pack for implementing data classification across your organization. Includes: standardized tag taxonomy (PII, PCI, confidential, public), business glossary terms, automated classification rules (regex, ML-based), and access policies by classification level. Integrates with DataHub, OpenMetadata, and native warehouse tagging.Sat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/resources/opa-data-access-policy/https://emanisha.in/de-agent/resources/opa-data-access-policy/A comprehensive OPA policy bundle for enforcing data access controls across your data platform. Includes policies for: row-level security, column masking, table-level permissions, and audit logging. Compatible with Trino, Presto, Snowflake, BigQuery, and Databricks via their OPA integrations.Sat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/resources/snowflake-warehouse-config/https://emanisha.in/de-agent/resources/snowflake-warehouse-config/A comprehensive Snowflake warehouse configuration template covering: warehouse sizing, auto-suspend/resume, resource monitors for cost control, query acceleration, and multi-cluster settings. Includes environments for dev, staging, and production with appropriate defaults.Sat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/references/a-duckdb-database-with-no-data-in-it/https://emanisha.in/de-agent/references/a-duckdb-database-with-no-data-in-it/A DuckDB database file does not have to hold any data. It can store nothing but view definitions that point at Parquet files on object storage, so a few hundred kilobytes behave like a shared catalog over a whole data lake. This post shows how to build one, host it, and attach itSat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/references/cost-effective-etl-with-duckdb-and-amazon-s3-tables-on-aws-glue/https://emanisha.in/de-agent/references/cost-effective-etl-with-duckdb-and-amazon-s3-tables-on-aws-glue/Learn how to pair DuckDB with AWS Glue 6.0 to run SQL-centric ETL on a single worker, reading Parquet from Amazon S3 and writing Apache Iceberg tables to Amazon S3 Tables. This post walks through a complete, runnable example and compares measured cost and runtime against an equivSat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/references/daily-life-as-a-jenkins-administrator-2/https://emanisha.in/de-agent/references/daily-life-as-a-jenkins-administrator-2/Jenkins Kubernetes setup As a team, we introduced Jenkins Kubernetes for every user a few months ago. Our goal was to make it easy to get started with Jenkins — no need for your own agent. We prepared guides on how to use it, which tools are already included in the image used bySat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/references/daily-life-as-a-jenkins-administrator/https://emanisha.in/de-agent/references/daily-life-as-a-jenkins-administrator/Agent ecosystem overview The second blog post focused on one specific part of your Kubernetes environment. I felt the urge to write more about it, so I thought a post about agents and managing them would be a good follow-up. Our agents include: 200 user-managed agents running onSat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/references/duckdb-and-hugging-face-querying-datasets-directly/https://emanisha.in/de-agent/references/duckdb-and-hugging-face-querying-datasets-directly/Hugging Face hosts hundreds of thousands of datasets, and DuckDB can read them directly, exactly where they are and without downloading anything, over the DuckDB `hf://` protocol. This post looks at how the integration works and the scenarios where it works best.Sat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/references/gsoc-2026-final-blog-how-it-was-to-be-a-contributor-at-jenkins/https://emanisha.in/de-agent/references/gsoc-2026-final-blog-how-it-was-to-be-a-contributor-at-jenkins/Hello everyone! I am Mallikarjun G D, and I have been a GSoC 2026 contributor for the project Continue: AI-Powered Chatbot for Quick Access to Jenkins Resources. Over the past few months, I have learnt a lot, not just about AI and Jenkins, but also about contributing to a large oSat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/references/importing-data-using-java-table-functions/https://emanisha.in/de-agent/references/importing-data-using-java-table-functions/When the analytics screen running on our main operational database got too slow, we moved a year of data into DuckDB on the same server. Getting the data in was the hard part. This is the story of every method we tried, and why a table function written in pure Java is the one weSat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/references/investigating-build-regressions-from-the-last-known-good-build/https://emanisha.in/de-agent/references/investigating-build-regressions-from-the-last-known-good-build/The following scenario is a case study. A Jenkins build fails in an integration test. The console contains a NoSuchMethodError, so the immediate task seems straightforward: find the incompatible dependency. Then someone points out that the last three builds failed too. Another deSat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/references/jenkins-and-ai-agents-the-trusted-engine-for-faster-software-del/https://emanisha.in/de-agent/references/jenkins-and-ai-agents-the-trusted-engine-for-faster-software-del/The 2026 Reality Check By 2026, software development has crossed a threshold. AI coding assistants are no longer just suggesting lines of code: agentic systems can take on larger tasks, refactor existing code, open pull requests, and attempt to fix defects with less hands-on direSat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/references/jev-and-duckdb-plain-english-conditions-in-sql/https://emanisha.in/de-agent/references/jev-and-duckdb-plain-english-conditions-in-sql/TypeSafe AI released Jev, a model that returns typed answers instead of text, on September 15, 2026. Within ten days, several community extensions let you filter, classify and score DuckDB rows with plain-English conditions. This post covers what Jev is, how the extensions work aSat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/references/kitfunso-hippo-memory/https://emanisha.in/de-agent/references/kitfunso-hippo-memory/Stop re-teaching your agent. Make your agent's memory work like a brain. Hippo is long-term memory for coding agents. It's a critical layer for your AI harness that connects across your different toolSat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/references/materialize-once-query-anywhere-introducing-iceberg-materialized/https://emanisha.in/de-agent/references/materialize-once-query-anywhere-introducing-iceberg-materialized/Amazon Redshift now supports Iceberg materialized views. Compute an aggregation once in Amazon Redshift and store the result as a standard Apache Iceberg table in Amazon S3, queryable by Amazon Athena, Apache Spark, Amazon SageMaker, and AWS Glue. This post covers use cases, incrSat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/references/monitoring-mwaa-orchestrated-etl-pipelines-with-amazon-opensearc/https://emanisha.in/de-agent/references/monitoring-mwaa-orchestrated-etl-pipelines-with-amazon-opensearc/Troubleshooting a multi-service, MWAA-orchestrated ETL pipeline often means hunting across scattered Amazon CloudWatch log groups. This post shows how to centralize your ETL logs in Amazon OpenSearch Service and query them in plain language through an MCP server on Amazon BedrockSat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/references/oaslananka-kicad-mcp-pro/https://emanisha.in/de-agent/references/oaslananka-kicad-mcp-pro/MCP server for KiCad, connecting AI agents to schematic and PCB workflows, ERC/DRC validation, DFM checks, BOM analysis, and manufacturing preparation—with controlled edits and human-gated release worSat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/references/optimize-consumer-rebalancing-on-amazon-msk-with-next-generation/https://emanisha.in/de-agent/references/optimize-consumer-rebalancing-on-amazon-msk-with-next-generation/The KIP-848 consumer protocol in Apache Kafka 4.0 redesigns consumer group rebalancing to eliminate stop-the-world pauses. This post explains how the consumer protocol works on Amazon MSK, how to enable it on MSK Standard and Express brokers, and how to diagnose and resolve slowSat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/references/query-across-accounts-and-table-formats-with-multi-catalog-in-am/https://emanisha.in/de-agent/references/query-across-accounts-and-table-formats-with-multi-catalog-in-am/Amazon EMR 8.1.0 adds multi-catalog support through the RedirectingSessionCatalog, so you can query Iceberg, Delta Lake, Hudi, and Hive tables through a single catalog and join data across AWS accounts without copying it. This post shows how to put these capabilities into practicSat, 24 Oct 2026 07:06:14 GMThttps://emanisha.in/de-agent/references/reading-ci-failures-across-pipelines-not-one-build-at-a-time/https://emanisha.in/de-agent/references/reading-ci-failures-across-pipelines-not-one-build-at-a-time/Jenkins is good at explaining a single build. The console, the stage view and the test report all answer "what happened in build #247". On a busy controller, most CI questions are not about a single build: Five pipelines are red. Is that five problems or one? Which of them brokeSat, 24 Oct 2026 07:06:14 GMT