Learning path
Spark / PySpark
Guided path · 1 stage · 14 resources
The guided path
Distributed dataframes: lazy DAGs, shuffles, skew, partitioning and physical plans. This path sequences the category's 14 resources by roadmap stage and composite score — work top-down inside each step.
Data Processing
- Spark Data Quality Review 41 — Audit a PySpark job for silent data-quality failures before it ships.
- Spark Skew Diagnosis 38.5 — Turn "one task is 40 minutes behind" into a named root cause and a fix.
- Spark job is slow 36.75 — Spark job is slow
- Spark Optimizer 34.5 — Spark Optimizer
- Start a PySpark project from the template 34.15 — Start a PySpark project from the template
- Aggregations and windows without the skew 34.15 — Aggregations and windows without the skew
8 more on the category page.
Prerequisites across the path
Documented prerequisites, deduplicated: skill-sql-review, skill-spark-data-quality-review.
What you will be able to do
- Identify the five classes of silent failure in a Spark transform
- Add pre/post row-count and null assertions to a job
- Recognise join fan-out caused by non-unique keys
- Read task-level duration histograms to confirm skew
- Choose between AQE skew join, salting and broadcast
- Reproduce skew locally with a bounded dataset
- Scaffold a PySpark project with the template
- Arrange transforms and offline test fixtures
Before you start
Security-flagged: Spark Data Quality Review, Spark Skew Diagnosis, Spark job is slow, Spark Optimizer, Spark Lens, PySpark Project Template — review the security posture first. Verification on these records: 14 community. Nothing here is independently audited — the trust scan and your own review are the controls.
Where this sits
Phase 1 of the roadmap. Stage context: Data Processing · side-by-side comparison · all learning topics