AWS

Cut Athena bytes scanned with partitions and projection

Walk one expensive Athena query through the Partition Advisor's levers in order: confirm the per-TB bill model, partition on the dominant filter column, replace MSCK REPAIR with partition projection, compact small files, and move to columnar compression.

Hold the cloud data stack mental model — storage, catalog, compute, IAM

Build the map before opening a console: object storage holds the bytes, a managed catalog describes them, and serverless compute bills for what a query scans.

Review a Glue job for sizing, bookmarks, and catalogue drift

Take a Glue/Spark job through the AWS Glue Job Reviewer's three checks: size workers from the input files' partitions instead of habit, verify the job bookmark against the input path layout, and diff the Glue Data Catalog schema against the Parquet file footer.

Stand up the S3 → Glue → Athena starter

Deploy the serverless query-in-place starter end to end: Terraform provisions buckets, catalog, and a bounded Athena workgroup, Parquet lands in a Hive layout, the table is registered with partition projection, and pruning is proven with explain analyze.

Wire cloud tooling — registry skills, MCP servers, and one review workflow

Wire the registry's cloud tooling into one agent workflow: install the Glue and Athena review skills, bind the AWS MCP server to a least-privilege profile, and compose both skills plus the MCP server into the architecture reviewer.

Capstone — the cloud guardrail checklist

Close the series by turning nine lessons into one standing checklist: workgroup bytes limits and compaction on storage, sized workers and stable bookmarks on compute, resource monitors and unit-metric alerts on spend, and least-privilege IAM everywhere — then produce the four review outputs as evidence.

Govern cloud spend — right-sizing, resource monitors, and unit economics

Govern cloud spend the way you govern data quality: right-size each warehouse per environment, put a resource monitor between the workload and the invoice, and alert on cost-per-run instead of total spend.

Incident — the job that quadrupled in cost overnight

An invoice alert or a cost-per-run dashboard spike lands: one job's compute cost quadrupled with no functional change.

Production gotchas — surprise bills, region defaults, and IAM overreach

The surprises that turn into invoices: workers sized by habit, a renamed prefix that silently resets a bookmark, thousands of small files, an agent holding an admin role, and a query that scans everything because the predicate missed the partition column.

Run an AWS data architecture review pass

Point the AWS Architecture Reviewer at an architecture description or IaC and follow its four-step workflow: map the services to a data flow, check IAM for least privilege, estimate dominant cost drivers, and identify single points of failure.

Athena Partition Advisor

• Read the scanned-bytes metric per query
• Apply partition projection for high-cardinality columns

AWS Glue Job Reviewer

• Size Glue workers from data volume, not habit
• Use job bookmarks correctly

AWS MCP Servers (Labs)

A suite of AWS Labs servers — including documentation search, CDK/Terraform assistance, and service-specific servers — for agents working on AWS.

AWS Architecture Reviewer

Review an AWS data architecture for cost, IAM and reliability risks.

Terraform AWS Data Platform

S3 buckets with lifecycle + encryption, Glue catalogue, IAM roles scoped to least privilege, and Athena workgroups with query limits.

S3 → Glue → Athena

A serverless, query-in-place pattern: land Parquet in S3, catalogue it with Glue, and query with Athena while keeping scanned bytes (and cost) low.

AWS Glue Documentation

complete reference for Glue jobs, crawlers, the Data Catalog, job bookmarks, DPU configuration, and Ray integration. The job bookmarks and catalog behavior sections are the source of the most common Glue surprises; the Spark UI integration pages explain how to debug slow jobs. Essential for our Glue review skills and CDC pipeline patterns.

Cost-effective ETL with DuckDB and Amazon S3 Tables on AWS Glue

Learn how to pair DuckDB with AWS Glue 6.0 to run SQL-centric ETL on a single worker, reading Parquet from Amazon S3 and writing Apache Iceberg tables to Amazon S3 Tables.

Materialize once, query anywhere: Introducing Iceberg materialized views in Amazon Redshift

Amazon Redshift now supports Iceberg materialized views.