architecture

DE//AGENT currently lists 24 resources under architecture: 15 patterns, 5 tutorials, 3 references. Everything below is derived from the live registry — counts, capabilities, verification state and security posture come from the curated records, never from marketing copy.

Also present: 1 agent.

Documented learning objectives across these resources:

• Separate infrastructure metrics from data signals — freshness, volume, schema, distribution — via Hold the observability mental model: infrastructure metrics versus data signals
• Place lineage and the catalog on the observability map, next to metrics and traces — via Hold the observability mental model: infrastructure metrics versus data signals
• State what each layer answers and which tool owns it — via Hold the observability mental model: infrastructure metrics versus data signals
• Explain why ordering and partitions behave the way they do — via Hold the streaming mental model: events, partitions, delivery semantics
• Treat at-least-once delivery and replays as the normal case — via Hold the streaming mental model: events, partitions, delivery semantics
• Choose between lambda, kappa, and plain batch for a latency and reprocessing need — via Hold the streaming mental model: events, partitions, delivery semantics
• Frame any data system as a tradeoff among reliability, scalability, and maintainability — via Hold the system design mental model — data systems as tradeoffs
• Name the six mechanisms DDIA teaches and map each one to a system you operate — via Hold the system design mental model — data systems as tradeoffs

7 more objectives live on the individual resource pages.

These resources map to stages: Observability (1), Streaming (4), System Design (4), Cloud (2), Warehousing (3), Data Processing (2), Lakehouse (2), Agentic Data Engineering (1), Data Quality (2), Reliability (2), Orchestration (1). Follow the four-phase roadmap to place this topic inside a progression instead of learning it in isolation.

Every entry, its shape and its trust state — sorted by the registry's composite score. This table is the honest version of a marketing comparison: no sponsored placements, no unverifiable claims.

12 more in the full list below.

How scores are computed: source quality score.

Highest composite score first:

1. Medallion Zones (raw / curated / serving) score 37.25 — beginner
2. Incremental Batch with a Watermark score 37.25 — intermediate
3. Quarantine then Publish score 37.25 — intermediate

Scores rank provenance, completeness and trust — read how verification works before adopting anything.

Start here if you're new: Medallion Zones (raw / curated / serving), dbt as Models + Tests + Docs, Idempotent Partition Overwrite, Hold the observability mental model: infrastructure metrics versus data signals.

Reach for these with production experience: Data Vault (hubs, links, satellites), Lambda vs Kappa (batch + stream), Safe Backfill.

Security-flagged entries: Postgres CDC to Lakehouse (2 flags), Streaming Dedup and Upsert (2 flags), Cost Guardrails (3 flags), AWS Architecture Reviewer (3 flags), Compaction & File-Size Management (3 flags). Flags mean the record declares capabilities like credential use or write access — read the security posture on each page before connecting.

Known limits: A single small pipeline with one consumer,When storage cost dominates and copies are wasteful (Medallion Zones (raw / curated / serving))

• Agent-in-the-Loop Review — Coding agents can do real data-engineering work, but can also ship a schema change or drop a table. details
• Postgres CDC to Lakehouse — Batch extraction from an OLTP database loads it and still misses deletes and mid-day changes. details
• Hold the observability mental model: infrastructure metrics versus data signals — Two layers live on one data platform: infrastructure metrics (CPU, lag, job success) and data signals (freshness, volume, schema, distribution). This lesson separates them, places … details
• Hold the streaming mental model: events, partitions, delivery semantics — Build the vocabulary before touching a cluster: events instead of rows, partitions instead of tables, at-least-once delivery instead of exactly-once, and the lambda/kappa decision … details

Adjacent topics: tutorial (5 shared), lakehouse (3 shared), streaming (3 shared), batch (3 shared), cost (2 shared), review (2 shared).

Study path: the roadmap places this topic in a progression · the scoring model explains the numbers · the trust rule catalog shows every security check these entries passed through.