Tune a slow job — memory, partitions, baselines

At a glance

Learning objectives

• Classify a slow stage into skew, partition count, spill, or scan
• Apply the single highest-impact fix with an expected effect
• Verify the change against baseline shuffle/spill/scan metrics

What you need

• The job's event log or Spark UI metrics
• A stored cost baseline for the job
• [object Object]

Steps

1. Parse task-duration and shuffle metrics. (from agent-spark-optimizer )
2. Classify the stage — skew, partition count, spill, or scan. (from agent-spark-optimizer )
3. Spill → increase partition count or raise executor memory. (from runbook-spark-job-slow )
4. Over-scan → add the partition predicate; confirm with bytes scanned. (from runbook-spark-job-slow )
5. Propose the single highest-impact change with expected effect, then verify against the baseline metrics. (from agent-spark-optimizer )
6. Baseline shuffle/spill/scan metrics and fail CI on regression, and add a partition filter lint on source reads. (from runbook-spark-job-slow )

Check yourself

• You can name which of skew, partition count, spill, or scan was the cause, with the metrics to prove it.
• The change you shipped is the single highest-impact one, and the stored baseline shows no regression.

Worked example

1. Parse task-duration and shuffle metrics.
2. Classify: skew, partition count, spill, or scan.
3. Propose the single highest-impact change with expected effect.
4. Verify against the baseline metrics.

Parses Spark event logs into a task histogram, flags skew and shuffle
amplification, and compares against a stored cost baseline.

- Baseline shuffle/spill/scan metrics and fail CI on regression.
- Add a partition filter lint on source reads.

Sourced from

• Spark Optimizer
• Spark job is slow
• Spark Lens

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• repartition(1) serialises the stage — never reach for it to 'fix' slowness.
• Spill shows up as slow tasks, not failures — check spill bytes before paying for bigger executors.
• AQE skew-join healing does not fix over-scan; check bytes scanned separately.

Production considerations

• Baseline shuffle/spill/scan metrics and fail CI on regression.
• Monitor task duration p99/p50 ratio, shuffle read/write bytes per run, spill bytes per stage.

Diagram

slow job
   |
   |- parse --> task-duration + shuffle metrics
   |
   |- classify --> skew | partition count | spill | scan
   |
   |- fix --> single highest-impact change
   |
   |- verify --> baseline metrics --> CI guard fails on regression

Registry stats