DE//AGENT
Explore
Project community intermediate

S3 → Glue → Athena

A serverless, query-in-place pattern: land Parquet in S3, catalogue it with Glue, and query with Athena while keeping scanned bytes (and cost) low.

At a glance

Where AWS · tech-s3 · tech-glue · tech-athena

Learning objectives

  • Lay out Parquet so Athena prunes well
  • Use partition projection instead of repeated crawls
  • Measure and cut bytes scanned

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Implementation

  • Write Parquet compressed, target 128 MB files.
  • Define the table with projection on dt (no MSCK REPAIR).
  • Query only the needed columns; compare scanned bytes before/after.

Tests

  • explain analyze shows partition pruning on dt
  • bytes scanned per query within budget

Failure Scenarios

  • A new partition is invisible → projection pattern doesn't match the layout
  • Query scans everything → predicate not on the partition column
  • Thousands of tiny files → compact

Production Hardening

  • Athena workgroup with a per-query bytes-scanned limit
  • A nightly compaction job
  • CloudWatch alarms on scanned bytes per hour

Registry stats

Verification community · Sep 22, 2026
Completeness 81
Trust multiplier ×0
Composite 30.15 · adequate
Freshness 32 days since last source check
Difficulty intermediate
Version 1.1.0
License MIT
Security flags 5

Repository Structure

s3-glue-athena/
  terraform/            # s3, glue catalog, athena workgroup
  ddl/table.sql         # partition projection DDL
  queries/*.sql
  data/dt=2026-09-01/*.parquet

Architecture

Parquet in S3 (Hive layout: dt=..) ─▶ Glue Data Catalog ─▶ Athena (serverless SQL) ▲ │ [Terraform: buckets, catalog, workgroup with bytes limit] └─▶ BI / notebooks

Setup

  • terraform init && terraform apply
  • aws s3 sync data/ s3://<bucket>/<prefix>/
  • run ddl/table.sql in Athena to register with projection

Sample Data

A small synthetic Parquet dataset partitioned by dt, with one deliberately oversized file to demonstrate compaction.