The batch processing mental model

At a glance

Learning objectives

• Name the four causes of a slow batch job: skew, partition count, spill, over-scan
• Explain what a shuffle does to a hot key
• State the idempotent-write invariant for re-runnable batch jobs

What you need

• A Spark job to inspect — its Spark UI stage view or event log
• The output table's partition layout (dt=... or equivalent)

Steps

1. Batch processing recomputes per logical partition: `INSERT OVERWRITE partition(dt=...)` or Delta/Iceberg replace-where replace that partition atomically. (from pattern-idempotent-partition-overwrite )
2. Never append to a partition you can recompute — the invariant is that running the same date twice yields identical output. (from pattern-idempotent-partition-overwrite )
3. A shuffle moves data by key, and key skew on a join or group-by key is the first cause of a slow stage. (from runbook-spark-job-slow )
4. The second cause is partition count: too few partitions after a filter, or too many after a shuffle. (from runbook-spark-job-slow )
5. The third cause is spill to disk from an oversized partition. (from runbook-spark-job-slow )
6. The fourth cause is reading far more data than needed (missing partition filter). (from runbook-spark-job-slow )
7. When a job skews or slows, check the config reference and the tuning guide before changing code. (from ref-spark-docs )

Check yourself

• You can name the four causes of a slow batch job: key skew, partition count, spill, over-scan.
• A re-run for dt=2026-09-01 overwrites that partition and produces identical output.

Worked example

- Key skew on a join or group-by key
- Too few partitions after a filter, or too many after a shuffle
- Spill to disk from an oversized partition
- Reading far more data than needed (missing partition filter)

INSERT OVERWRITE partition(dt=...)   # or Delta/Iceberg replace-where

The invariant: running the same date twice yields identical output.

Spark SQL, the DataFrame API, tuning and the configuration catalogue,
versioned per Spark release. The config reference alone resolves most
tuning debates.

Sourced from

• Spark job is slow
• Idempotent Partition Overwrite
• Apache Spark Documentation

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Too few partitions after a filter, or too many after a shuffle: both look like 'Spark is slow'.
• Spill shows up as slow tasks, not failures — check spill bytes before paying for bigger executors.
• An empty partition and an absent partition both read as 'no data' downstream.

Production considerations

• Baseline shuffle/spill/scan metrics per run so regressions are visible before users feel them.
• Store the logical date explicitly as a partition column, not derived at read time.

Diagram

batch job
   |
   |- read --> partitions (unit of parallelism)
   |
   |- shuffle by key --> hot key --> skew
   |        > oversized partition --> spill to disk
   |
   |- scan --> missing partition filter --> over-scan
   |
   |- write --> INSERT OVERWRITE dt=... --> same date twice = same output

Registry stats