DE//AGENT
Explore
Project community intermediate

API → S3 → Spark → Parquet

Land a paginated REST API into S3 raw, transform with PySpark into partitioned Parquet, and make the whole thing idempotent and testable offline.

At a glance

Where APIs & Ingestion · tech-python · tech-s3 · tech-spark

Learning objectives

  • Handle pagination, retries and rate limits correctly
  • Write an idempotent partition overwrite
  • Test a Spark transform without a cluster

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Implementation

  • Extract with an explicit cursor and a max-pages guard; log every request.
  • Land raw JSON untouched, keyed by ingestion date.
  • [object Object]
  • Write Parquet with INSERT OVERWRITE on dt so re-runs are safe.

Tests

  • row count stable across two runs (idempotency)
  • no nulls in the business key
  • partition dt matches the logical date, not the wall clock

Failure Scenarios

  • API returns 429 → backoff then resume from the last cursor, not from page 1
  • A page arrives with a changed field type → quarantine the page, alert
  • A re-run for the same date duplicates rows → the overwrite is not partition-scoped

Production Hardening

  • Retry with jittered exponential backoff
  • Alert on extract lag and on null business keys
  • Checksum raw files; store the API's cursor for replay

Registry stats

Verification community · Sep 28, 2026
Completeness 81
Trust multiplier ×0
Composite 32.15 · adequate
Freshness 26 days since last source check
Difficulty intermediate
Version 1.2.0
License MIT
Security flags 5

Repository Structure

api-s3-spark-parquet/
  extract/api_client.py     # pagination, backoff, idempotency key
  transform/orders.py       # typed, deduped, partitioned
  extract/land_raw.py       # writes raw JSON, never overwrites silently
  tests/data/*.json         # tiny fixtures
  tests/test_orders.py
  docker-compose.yml        # minio + spark local
  Makefile

Architecture

REST API ─▶ [extract: paginate + retry] ─▶ S3 raw/ (JSON, immutable) │ ▼ [PySpark transform: type, dedupe, partition] │ ▼ S3 curated/ (Parquet, dt=.., Hive layout)

Setup

  • make up # starts MinIO + a local Spark
  • make seed # uploads sample raw JSON
  • make test # runs the transform tests offline

Sample Data

tests/data/orders_page_1.json, tests/data/orders_page_2.json (3-5 rows each)