DE//AGENT
Explore
Tutorial Lesson 1 of 10 · The Cloud Series community beginner

Hold the cloud data stack mental model — storage, catalog, compute, IAM

Build the map before opening a console: object storage holds the bytes, a managed catalog describes them, and serverless compute bills for what a query scans. Walk the same shape through five managed engines — Glue, Athena, BigQuery, Snowflake, Databricks — and note the IAM surface each one needs before the first query.

At a glance

When You are about to open a cloud console for the first time and need the map before the menus; A stack mixes engines (Glue, Athena, BigQuery, Snowflake, Databricks) and one vocabulary has to cover all of them
Where AWS
Why Every managed service rents you the same shape — storage, catalog, compute, IAM — so learn the shape once and every console becomes familiar.

Learning objectives

  • Explain storage/compute separation with one managed example per layer
  • Name the IAM surface each managed service needs before the first query
  • Map any cloud data stack into one data flow — storage, catalog, compute, serve

What you need

  • No cloud account required — this is a concepts lesson
  • The AWS Glue, BigQuery, and Snowflake documentation pages open beside you

Steps

  1. See the stack as Terraform sees it — S3 buckets with lifecycle + encryption, a Glue catalogue, IAM roles scoped to least privilege, and Athena workgroups with query limits. (from tpl-terraform-aws-data )
  2. Read the flow in one line — Parquet in S3 (Hive layout, dt=..) into the Glue Data Catalog, queried by Athena serverless SQL, with Terraform provisioning buckets, catalog, and a workgroup with a bytes limit. (from project-s3-glue-athena )
  3. Learn the managed Glue surfaces — jobs, crawlers, the Data Catalog, job bookmarks, DPU configuration, and Ray integration are what every Glue pipeline touches. (from ref-glue-docs )
  4. Know where the surprises live — job bookmarks and the catalogue's behaviour are the source of the most common Glue surprises. (from ref-glue-docs )
  5. See storage/compute separation in BigQuery — partitioning, clustering, slot management, and the query optimizer's behavior are the levers, and the partitioning/clustering pages are the exact levers for cost control. (from ref-bigquery-docs )
  6. See it again in Snowflake — virtual warehouses, micro-partitions, clustering keys, and zero-copy cloning are the managed primitives; the performance and cost sections are the reference for clustering-key and warehouse-sizing decisions. (from ref-snowflake-docs )
  7. Close the map with Databricks — Delta Live Tables, Unity Catalog, liquid clustering, and the Photon engine are platform-specific and only documented there. (from ref-databricks-docs )

Check yourself

  • You can draw the S3 → Glue Data Catalog → Athena flow from memory and say which layer stores bytes, which describes them, and which bills for scanning.
  • Asked what each engine bills for compute, you name the unit — DPU configuration for Glue, slot management for BigQuery, credits/hour warehouse size for Snowflake — without opening a pricing page.
  • For each managed service you can name the IAM surface — the least-privilege role or profile the job or agent assumes.

Worked example

The stack in one line — from project-s3-glue-athena 

Parquet in S3 (Hive layout: dt=..) ─▶ Glue Data Catalog ─▶ Athena (serverless SQL)
          ▲                                                    │
[Terraform: buckets, catalog, workgroup with bytes limit]    └─▶ BI

Cost levers in BigQuery — from ref-bigquery-docs 

BigQuery levers: partitioning, clustering, slot management, query optimizer
behavior. The partitioning/clustering pages are the exact levers for cost
control; the query optimizer section explains slot utilization.

The IaC baseline — from tpl-terraform-aws-data 

S3 buckets with lifecycle + encryption, Glue catalogue, IAM roles scoped to
least privilege, and Athena workgroups with query limits.

Sourced from

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

  • Job bookmarks and the catalogue's behaviour are the source of the most common Glue surprises — read those doc sections before your first job.
  • Liquid clustering and Unity Catalog are platform-specific and only documented in the Databricks docs — engine assumptions do not transfer.
  • IAM roles scoped to least privilege are part of the stack definition, not a bolt-on after the first query.

Production considerations

  • Keep the Terraform template's shape as your baseline — buckets with lifecycle + encryption, catalog, least-privilege roles, workgroup with query limits.
  • Re-read each engine's performance and cost doc section when a sizing decision stalls; the levers are documented.

Diagram

cloud data stack (one shape, five engines)
   |
   |- storage: object storage (S3) holds bytes; lifecycle + encryption live here
   |
   |- catalog: Glue Data Catalog describes the bytes
   |
   |- compute: DPUs (Glue) | scanned bytes (Athena) | slots (BigQuery) | credits/hour (Snowflake)
   |
   |- IAM: least-privilege roles/profiles gate every layer
   |
    --> map services to one data flow before judging any of it

Registry stats

Verification community · Oct 23, 2026
Completeness 81
Trust multiplier ×1.25
Composite 34.15 · adequate
Freshness 1 days since last source check
Difficulty beginner
Version 0.1.0
License CC-BY-4.0
Security flags none