DE//AGENT
Explore
Tutorial Lesson 2 of 10 · The Cloud Series community beginner

Stand up the S3 → Glue → Athena starter

Deploy the serverless query-in-place starter end to end: Terraform provisions buckets, catalog, and a bounded Athena workgroup, Parquet lands in a Hive layout, the table is registered with partition projection, and pruning is proven with explain analyze. The sample data ships one deliberately oversized file to demonstrate compaction.

At a glance

When A serverless lakehouse on AWS is needed without a cluster to run; Athena cost must stay bounded while dashboards query S3 Parquet
Where AWS
Why The starter makes the query-in-place pattern real with projection DDL and a workgroup bytes limit, so cost and layout are right from day one.

Learning objectives

  • Lay out Parquet so Athena prunes well
  • Use partition projection instead of repeated crawls
  • Measure and cut bytes scanned

What you need

  • Terraform and an AWS profile it can apply with
  • The starter repo (terraform/, ddl/table.sql, queries/, sample Parquet under data/dt=)
  • [object Object]

Steps

  1. Provision the stack: terraform init && terraform apply. (from project-s3-glue-athena )
  2. Land the data: aws s3 sync data/ s3://<bucket>/<prefix>/. (from project-s3-glue-athena )
  3. Register the table: run ddl/table.sql in Athena to register with projection. (from project-s3-glue-athena )
  4. Define the table with projection on dt (no MSCK REPAIR). (from project-s3-glue-athena )
  5. Verify pruning: explain analyze shows partition pruning on dt. (from project-s3-glue-athena )
  6. Add the workgroup guardrail — an Athena workgroup with a per-query bytes-scanned limit. (from project-s3-glue-athena )

Check yourself

  • explain analyze on a dt-filtered query shows partition pruning, and the workgroup reports scanned bytes within budget.
  • A query that scans everything is traced to a predicate that is not on the partition column, then fixed without touching the layout.

Worked example

Provision and land the data — from project-s3-glue-athena 

terraform init && terraform apply
aws s3 sync data/ s3://<bucket>/<prefix>/

Starter repo layout — from project-s3-glue-athena 

s3-glue-athena/
  terraform/            # s3, glue catalog, athena workgroup
  ddl/table.sql         # partition projection DDL
  queries/*.sql
  data/dt=2026-09-01/*.parquet

Scaffold the stack from the Terraform template — from tpl-terraform-aws-data 

amp template new de-agent/terraform-aws-data-platform ./infra

Sourced from

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

  • A new partition is invisible → the projection pattern does not match the layout.
  • A query scans everything → the predicate is not on the partition column.
  • Thousands of tiny files → compact; the sample data ships one deliberately oversized file to demonstrate the fix.

Production considerations

  • Run the workgroup with a per-query bytes-scanned limit and CloudWatch alarms on scanned bytes per hour.
  • Add a nightly compaction job to keep the Parquet layout near the 128 MB target.

Diagram

s3 -> glue -> athena starter
   |
   |- terraform apply --> buckets + catalog + workgroup (bytes limit)
   |
   |- s3 sync data/ --> hive layout dt=..
   |
   |- ddl/table.sql --> partition projection (no MSCK REPAIR)
   |
    |- explain analyze --> pruning on dt --> bytes scanned in budget

Registry stats

Verification community · Oct 21, 2026
Completeness 81
Trust multiplier ×1.25
Composite 34.15 · adequate
Freshness 3 days since last source check
Difficulty beginner
Version 0.1.0
License CC-BY-4.0
Security flags none