DE//AGENT
Explore
Tutorial Lesson 8 of 10 · The Cloud Series community intermediate

Production gotchas — surprise bills, region defaults, and IAM overreach

The surprises that turn into invoices: workers sized by habit, a renamed prefix that silently resets a bookmark, thousands of small files, an agent holding an admin role, and a query that scans everything because the predicate missed the partition column. Each gotcha names the metric that catches it early.

At a glance

When A Glue job's bill grew without any change in data volume; An agent was granted broad credentials as a temporary measure; A new region or profile default may be silently routing resources
Where AWS
Why Cloud bills are silent until the invoice — every gotcha here has a metric or a diff that catches it the day it lands.

Learning objectives

  • Catch the Glue cost leaks before the invoice — oversized workers, reset bookmarks, drifted catalogues
  • Keep small files and repeated metadata repairs out of the bill
  • Bind agents to least-privilege profiles so IAM overreach and region drift stay impossible

What you need

  • Recent Glue task histograms and Athena scanned-bytes metrics
  • [object Object]

Steps

  1. Watch for the most common Glue cost leak — over-provisioned workers; size from data volume, not habit. (from skill-aws-glue-reviewer )
  2. Check the input path layout before every bookmark-dependent run — a renamed prefix silently resets the bookmark into a full reprocess. (from skill-aws-glue-reviewer )
  3. Compact small files — per-file overhead dominates at under 16 MB, so small files inflate the bill even when pruning works. (from skill-athena-partition-advisor )
  4. Recognize the classic combo the reviewer flags — a Glue job running with an admin role and no bookmark, causing nightly full reprocessing. (from agent-aws-architecture-reviewer )
  5. Never attach an admin role to an agent — bind to a dedicated AWS profile with least privilege and short-lived SSO credentials. (from mcp-aws )
  6. Set AWS_REGION deliberately — it is the default region for the AWS MCP servers, while AWS_PROFILE decides what is reachable. (from mcp-aws )
  7. Treat every BigQuery MCP query as billable — grant bigquery.dataViewer + bigquery.jobUser only, set a custom quota, and consider maximum_bytes_billed on every job. (from mcp-bigquery )

Check yourself

  • The task histogram shows balanced tasks after re-sizing, and no Glue job in the account runs with an admin role and no bookmark.
  • Tables averaging under 16 MB per file are compacted, and the per-query bill no longer carries per-file overhead.
  • Agents bind to a dedicated least-privilege profile with short-lived SSO credentials and an explicit AWS_REGION.

Worked example

The Glue review checklist — from skill-aws-glue-reviewer 

- Start worker sizing at 2x the number of input files' partitions; watch the task histogram.
- A bookmark only helps if the input path layout is stable.
- A renamed prefix silently resets the bookmark → full reprocess.
- Diff the Glue Data Catalog schema against the file footer.

The billable surface — from mcp-bigquery 

Queries are billable. Grant `bigquery.dataViewer` + `bigquery.jobUser` only,
set a custom quota, and consider `maximum_bytes_billed` on every job.

Agent credential binding — from mcp-aws 

Bind to a dedicated AWS profile with least privilege, use short-lived SSO
credentials, and never attach an admin role. Treat resource-creating tools
as requiring human confirmation.

Sourced from

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

  • A renamed prefix under the input path silently resets the bookmark into a full reprocess.
  • Per-file overhead dominates at under 16 MB — small files inflate the bill even when pruning works.
  • A Glue job running with an admin role and no bookmark causes nightly full reprocessing — the reviewer's own worked example.

Production considerations

  • Keep the task histogram and the scanned-bytes metric in every review; both are the fastest signals that sizing or layout is wrong.
  • Treat resource-creating MCP tools as requiring human confirmation, and rotate short-lived SSO credentials on a schedule.

Diagram

cloud production gotchas
   |
   |- workers sized by habit --> over-provisioned --> most common Glue cost leak
   |
   |- renamed prefix --> bookmark silently resets --> full reprocess
   |
   |- files < 16 MB --> per-file overhead dominates --> compact
   |
   |- admin role on a job --> nightly full reprocess + IAM overreach
   |
   |- AWS_REGION unset --> resources land in the wrong default region
   |
    --> billable queries (BigQuery): custom quota + maximum_bytes_billed

Registry stats

Verification community · Oct 23, 2026
Completeness 81
Trust multiplier ×1.25
Composite 34.15 · adequate
Freshness 1 days since last source check
Difficulty intermediate
Version 0.1.0
License CC-BY-4.0
Security flags none