Apache Spark

DE//AGENT currently lists 13 resources under Apache Spark: 5 skills, 4 projects, 3 templates. Everything below is derived from the live registry — counts, capabilities, verification state and security posture come from the curated records, never from marketing copy.

Also present: 1 rule pack.

Documented learning objectives across these resources:

• Identify the five classes of silent failure in a Spark transform — via Spark Data Quality Review
• Add pre/post row-count and null assertions to a job — via Spark Data Quality Review
• Recognise join fan-out caused by non-unique keys — via Spark Data Quality Review
• Configure SQLFluff for your warehouse dialect — via SQLFluff Core Rule Pack
• Customize rules for team standards — via SQLFluff Core Rule Pack
• Integrate SQLFluff into CI/CD pipelines — via SQLFluff Core Rule Pack
• Fix legacy SQL debt incrementally — via SQLFluff Core Rule Pack
• Scope a backfill by date and by dependency — via Backfill Planner

23 more objectives live on the individual resource pages.

These resources map to stages: Python + SQL (1), Cloud (2), Data Processing (4), Orchestration (1), Reliability (1), Streaming (3), Data Quality (1). Follow the four-phase roadmap to place this topic inside a progression instead of learning it in isolation.

Every entry, its shape and its trust state — sorted by the registry's composite score. This table is the honest version of a marketing comparison: no sponsored placements, no unverifiable claims.

1 more in the full list below.

How scores are computed: source quality score.

Highest composite score first:

1. Spark Data Quality Review score 41 — intermediate
2. SQLFluff Core Rule Pack score 39.7 — beginner
3. Backfill Planner score 39.5 — advanced

Scores rank provenance, completeness and trust — read how verification works before adopting anything.

Start here if you're new: SQLFluff Core Rule Pack, Local Docker Data Stack.

Reach for these with production experience: Backfill Planner, Spark Skew Diagnosis, Spark Cost Regression Guard.

Security-flagged entries: Spark Data Quality Review (1 flag), SQLFluff Core Rule Pack (1 flag), Backfill Planner (1 flag), Pipeline → DQ → quarantine → publish (2 flags), Spark Skew Diagnosis (2 flags), AWS Glue Job Reviewer (2 flags), Spark Cost Regression Guard (2 flags), Local Docker Data Stack (2 flags), PySpark Project Template (2 flags), Postgres → CDC → Kafka → Lakehouse (4 flags), API → S3 → Spark → Parquet (5 flags), Kafka → Spark Streaming → Delta (4 flags), Kafka Streaming Job Template (4 flags). Flags mean the record declares capabilities like credential use or write access — read the security posture on each page before connecting.

• Spark Data Quality Review — A review skill an agent runs against a Spark/PySpark transform. It checks null propagation, join fan-out, silent type coercion, non-deterministic row counts and missing post-write … details
• SQLFluff Core Rule Pack — A curated SQLFluff configuration with rules optimized for modern data engineering workflows. Includes dialect-specific rules for dbt (Jinja), Spark SQL, Snowflake, BigQuery, and Re… details
• Backfill Planner — Produces a backfill plan: date ranges, blast radius, resource contention, idempotency check, validation gates and rollback. details
• Pipeline → DQ → quarantine → publish — Put a contract in front of a table: validate each batch, publish the clean rows, quarantine the rest with reasons, and alert on the quarantine rate. details

Real install commands from the registry — read the source before running any of them:

npx de-agent-platform add spark-data-quality-review

Spark Data Quality Review — review first

pip install sqlfluff==3.0.0 sqlfluff-templater-dbt

SQLFluff Core Rule Pack — review first

npx de-agent-platform add backfill-planner

Backfill Planner — review first

npx de-agent-platform add spark-skew-diagnosis

Spark Skew Diagnosis — review first

Official source: Apache Spark (license: Apache-2.0) — start there for the canonical documentation, then come back for the vetted registry entries.

Adjacent topics: Apache Kafka (4 shared), PySpark (3 shared), Delta Lake (3 shared), Docker (3 shared), template (3 shared), Python (2 shared).

Study path: the roadmap places this topic in a progression · the scoring model explains the numbers · the trust rule catalog shows every security check these entries passed through.