Series · roadmap stage 8/13
The Cloud Series
Ten lessons in order, from the mental model to a capstone checklist. Every lesson is structured by us and grounded in a real, cited source from the registry — no invented commands.
10 of 10 lessons published
- Lesson 1 Hold the cloud data stack mental model — storage, catalog, compute, IAM
Build the map before opening a console: object storage holds the bytes, a managed catalog describes them, and serverless compute bills for what a query scans. Walk the same shape through five managed engines — Glue, Athena, BigQuery, Snowfl
- Lesson 2 Stand up the S3 → Glue → Athena starter
Deploy the serverless query-in-place starter end to end: Terraform provisions buckets, catalog, and a bounded Athena workgroup, Parquet lands in a Hive layout, the table is registered with partition projection, and pruning is proven with ex
- Lesson 3 Review a Glue job for sizing, bookmarks, and catalogue drift
Take a Glue/Spark job through the AWS Glue Job Reviewer's three checks: size workers from the input files' partitions instead of habit, verify the job bookmark against the input path layout, and diff the Glue Data Catalog schema against the
- Lesson 4 Cut Athena bytes scanned with partitions and projection
Walk one expensive Athena query through the Partition Advisor's levers in order: confirm the per-TB bill model, partition on the dominant filter column, replace MSCK REPAIR with partition projection, compact small files, and move to columna
- Lesson 5 Govern cloud spend — right-sizing, resource monitors, and unit economics
Govern cloud spend the way you govern data quality: right-size each warehouse per environment, put a resource monitor between the workload and the invoice, and alert on cost-per-run instead of total spend. Ends with the unit-economics rule
- Lesson 6 Run an AWS data architecture review pass
Point the AWS Architecture Reviewer at an architecture description or IaC and follow its four-step workflow: map the services to a data flow, check IAM for least privilege, estimate dominant cost drivers, and identify single points of failu
- Lesson 7 Wire cloud tooling — registry skills, MCP servers, and one review workflow
Wire the registry's cloud tooling into one agent workflow: install the Glue and Athena review skills, bind the AWS MCP server to a least-privilege profile, and compose both skills plus the MCP server into the architecture reviewer. The same
- Lesson 8 Production gotchas — surprise bills, region defaults, and IAM overreach
The surprises that turn into invoices: workers sized by habit, a renamed prefix that silently resets a bookmark, thousands of small files, an agent holding an admin role, and a query that scans everything because the predicate missed the pa
- Lesson 9 Incident — the job that quadrupled in cost overnight
An invoice alert or a cost-per-run dashboard spike lands: one job's compute cost quadrupled with no functional change. Work the runbook — diff shuffle write and input bytes run-over-run, rank the four usual suspects, fix per cause, and cap
- Lesson 10 Capstone — the cloud guardrail checklist
Close the series by turning nine lessons into one standing checklist: workgroup bytes limits and compaction on storage, sized workers and stable bookmarks on compute, resource monitors and unit-metric alerts on spend, and least-privilege IA