The 3am batch incident path

At a glance

Learning objectives

• Collect the four pieces of evidence a failing Spark run needs
• Decide between revert, a config change, and a code fix
• Record the skew/spill/scan signature for the cost-per-run history

What you need

• The affected run's event log and Spark UI stage summary
• The job source and its recent config changes
• [object Object]

Steps

1. Symptoms — job duration rose sharply with no code change, one or a few tasks dominate the stage, and executors idle while a handful run long. (from runbook-spark-job-slow )
2. Collect the evidence — Spark UI stage summary p50/p99 task durations, shuffle read/write and spill bytes per task, input bytes scanned vs the expected partition size, and the run's event log. (from runbook-spark-job-slow )
3. Skew if p99 >> p50 (10x+), uniform slowness if p50 is high; check input bytes scanned vs table size. (from runbook-spark-job-slow )
4. No code rollback — the job is unchanged. Revert only if a config change (AQE flags, executor sizing) preceded the regression. (from runbook-spark-job-slow )
5. Fix — skew gets AQE skew join or a salted hot key; spill gets more partitions or executor memory; over-scan gets a partition predicate. Never repartition(1). (from runbook-spark-job-slow )
6. Page the on-call data engineer if the job sits on a golden-hour SLA; otherwise file it and let the owning team run diagnosis in hours. (from runbook-spark-job-slow )
7. Follow up — record the skew/spill/scan signature in the run's cost-per-run history, and add the partition-filter lint or CI metric guard that would have caught it. (from runbook-spark-job-slow )

Check yourself

• The evidence packet is attached: p50/p99 durations, shuffle/spill bytes per task, input bytes scanned, and the event log.
• The decision — revert vs AQE/salt/spill fix — cites a config change or a metric, not a hunch.

Worked example

- Spark UI stage summary: p50/p99 task durations for the slow stages
- Shuffle read/write and spill bytes per task
- Input bytes scanned vs the expected partition size
- The job's event log for the affected run

No code rollback — the job is unchanged. Revert only if a config change
(AQE flags, executor sizing) preceded the regression.

Page the on-call data engineer if the job sits on a golden-hour SLA;
otherwise file it and let the owning team run diagnosis in hours.

amp plugin add de-agent/spark-lens
/spark-analyze        # task histogram; flags skew and shuffle amplification

Sourced from

• Spark job is slow
• Spark Lens

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Spill shows up as slow tasks, not failures — check spill bytes before paying for bigger executors.
• repartition(1) serialises the stage — never reach for it to 'fix' slowness.
• p99 much greater than p50 means skew; uniformly high p50 means capacity. Salting helps the first, not the second.

Production considerations

• Monitor task duration p99/p50 ratio, shuffle read/write bytes per run, spill bytes per stage.
• Baseline shuffle/spill/scan metrics so CI fails a regression before the SLA does.

Diagram

3am incident
   |
   |- symptoms --> runtime jumped, no code change, few tasks dominate
   |
   |- evidence --> p50/p99 + shuffle/spill bytes + bytes scanned + event log
   |
   |- decide --> config change? revert : AQE / salt / partitions / predicate
   |
   |- escalate --> golden-hour SLA? page on-call
   |
   |- follow up --> record signature, add the CI guard

Registry stats