Skill community advanced
Spark Skew Diagnosis
Turn "one task is 40 minutes behind" into a named root cause and a fix.
Walks an agent through reading the Spark UI/event log to distinguish key skew, partition skew, and shuffle-write amplification, then selects between salting, AQE skew join, broadcast, and repartition strategies.
npx de-agent-platform add spark-skew-diagnosisAt a glance
| What | Turn "one task is 40 minutes behind" into a named root cause and a fix. |
|---|---|
| Where | Spark / PySpark · tech-spark |
Learning objectives
- Read task-level duration histograms to confirm skew
- Choose between AQE skew join, salting and broadcast
- Reproduce skew locally with a bounded dataset
Security scan
Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.
Details
Spark Skew Diagnosis
Signals
- Median task duration ~p50, but p99 is 30x+ → key skew.
- All tasks slow and shuffle read is huge → partition/scan problem, not skew.
- One executor at 100% while others idle → hot partition.
Decision order
- Confirm with
spark.sql.adaptive.skewJoin.enabled=true— if it fixes it, ship that. - If not, salt the hot key on both sides and aggregate twice.
- Broadcast only if the small side is < 100 MB after filtering.
- Never
repartition(1)to "fix" skew; it serialises the stage.
Evidence to attach
The task-duration histogram, the shuffle read/write bytes, and the top-10 keys by row count — the review is not complete without them.
Registry stats
| Verification | community · Sep 18, 2026 |
|---|---|
| Completeness | 100 |
| Trust multiplier | ×0.75 |
| Composite | 38.5 · adequate |
| Freshness | 36 days since last source check |
| Difficulty | advanced |
| Version | 0.1.4 |
| License | Apache-2.0 |
| Security flags | 2 |
Did this solve your problem?
Spark Skew Diagnosis