Roadmap · stage 11/13

Reliability

Stage 11 of 13 · 31 resources · 6 categories

What this stage is

Reliability is stage 11 of 13 in the DE//AGENT roadmap. It currently holds 31 resources: 10 tutorials, 5 skills, 5 hooks across 6 categories.

What it teaches

SLOs, incidents, cost control and safe change.

What you can do after this stage

Write and run a runbook
Respond to an incident and prevent a repeat

Exit gate

Gate: You can operate a data system for years. The stage deliverable: runbook-spark-job-slow.

Gates are self-assessed — the roadmap does not track your progress; it tells you what to be able to do before moving on.

Categories in this stage

Security (3), Governance (4), Cost Engineering (11), Reliability (6), Testing (1), Production Operations (6)

Registry resources for this stage

Sorted by composite score — provenance, completeness and trust, per the scoring model:

Resource	Type	Difficulty	Verification	Score
Data Classification & Tagging Governance Pack	Governance Pack	intermediate	community	40.7
OPA Data Access Control Policy	Policy	intermediate	community	39.35
Incident Postmortem Writer	Skill	beginner	community	38.9
Pipeline Test Author	Skill	intermediate	community	38
Spark Cost Regression Guard	Skill	advanced	community	37
Spark job suddenly costs 4x	Runbook	advanced	community	36.75
Runbook Runner	Plugin	beginner	community	35.6
PII Classifier	Skill	intermediate	community	34.4
Compute unit economics for a pipeline	Tutorial	beginner	community	34.15
Install cost guardrails before the invoice	Tutorial	beginner	community	34.15
The data reliability mental model — freshness, completeness, correctness	Tutorial	beginner	community	34.15
Attribute a 4x Spark cost regression	Tutorial	intermediate	community	34.15
Attribute spend with the Cost Lens plugin	Tutorial	intermediate	community	34.15
Capstone — the reliability checklist for a pipeline you can page on	Tutorial	intermediate	community	34.15
DR and replication — design the recovery path before you need it	Tutorial	intermediate	community	34.15
Idempotency and reruns — make every task safe to run twice	Tutorial	intermediate	community	34.15
Plan and execute a backfill that doesn't break production	Tutorial	intermediate	community	34.15
Production gotchas — silent failures and partial writes	Tutorial	intermediate	community	34.15
Destructive SQL Guard	Hook	beginner	community	33.6
Production Write Confirmation	Hook	beginner	community	33.6
Secret Scan on Commit	Hook	beginner	community	32.1
PII Leak Guard	Hook	advanced	community	31.1
Spark Cost Check	Hook	advanced	community	31.1
Cost Unit-Economics Reporter	Skill	intermediate	community	30.4
Cost Guardrails	Pattern	intermediate	community	30.25
Incident Assistant	Agent	intermediate	community	30
Pipeline failed	Runbook	beginner	community	28.75
Data Mesh (O'Reilly)	Reference	—	community	26.7
Cost Lens	Plugin	advanced	community	25.6
Cost Optimizer	Agent	advanced	community	24.5
Reading CI Failures Across Pipelines, Not One Build at a Time	Reference	—	community	23.55

Documented learning objectives

Implement consistent data classification taxonomy
Build business glossary with ownership
Automate classification via regex and ML
Enforce access policies by classification level
Understand OPA/Rego for data access control
Implement row-level security policies
Apply column masking for PII
Audit data access decisions

Continue the progression

← Observability · System Design →

← Full roadmap · All learning topics