PySpark

DE//AGENT currently lists 3 resources under PySpark: 1 skill, 1 template, 1 project. Everything below is derived from the live registry — counts, capabilities, verification state and security posture come from the curated records, never from marketing copy.

Documented learning objectives across these resources:

• Identify the five classes of silent failure in a Spark transform — via Spark Data Quality Review
• Add pre/post row-count and null assertions to a job — via Spark Data Quality Review
• Recognise join fan-out caused by non-unique keys — via Spark Data Quality Review
• Handle pagination, retries and rate limits correctly — via API → S3 → Spark → Parquet
• Write an idempotent partition overwrite — via API → S3 → Spark → Parquet
• Test a Spark transform without a cluster — via API → S3 → Spark → Parquet

These resources map to stage: Data Processing (3). Follow the four-phase roadmap to place this topic inside a progression instead of learning it in isolation.

Every entry, its shape and its trust state — sorted by the registry's composite score. This table is the honest version of a marketing comparison: no sponsored placements, no unverifiable claims.

How scores are computed: source quality score.

Highest composite score first:

1. Spark Data Quality Review score 41 — intermediate
2. PySpark Project Template score 34.1 — intermediate
3. API → S3 → Spark → Parquet score 32.15 — intermediate

Scores rank provenance, completeness and trust — read how verification works before adopting anything.

Security-flagged entries: Spark Data Quality Review (1 flag), PySpark Project Template (2 flags), API → S3 → Spark → Parquet (5 flags). Flags mean the record declares capabilities like credential use or write access — read the security posture on each page before connecting.

• Spark Data Quality Review — A review skill an agent runs against a Spark/PySpark transform. It checks null propagation, join fan-out, silent type coercion, non-deterministic row counts and missing post-write … details
• PySpark Project Template — A testable PySpark project: src/transforms, tests with a local Spark session, a Dockerfile, and a Makefile that runs lint + tests offline. details
• API → S3 → Spark → Parquet — Land a paginated REST API into S3 raw, transform with PySpark into partitioned Parquet, and make the whole thing idempotent and testable offline. details

Real install commands from the registry — read the source before running any of them:

npx de-agent-platform add spark-data-quality-review

Spark Data Quality Review — review first

amp template new de-agent/pyspark-project ./my-pipeline

PySpark Project Template — review first

Official source: PySpark (license: Apache-2.0) — start there for the canonical documentation, then come back for the vetted registry entries.

Adjacent topics: Apache Spark (3 shared), Docker (2 shared).

Study path: the roadmap places this topic in a progression · the scoring model explains the numbers · the trust rule catalog shows every security check these entries passed through.