Incident — the job that quadrupled in cost overnight

At a glance

Learning objectives

• Attribute a cost spike to a stage with run-over-run byte diffs
• Tell apart fan-out, lost filters, and small-file regressions
• Cap the damage with resource monitors and unit-metric alarms

What you need

• Spark event logs or job metrics for the run before and the run during the spike
• [object Object]

Steps

1. Name the incident — one job's compute cost quadrupled without an obvious functional change; attribute it and reverse the regression. (from runbook-spark-job-suddenly-costs-4x )
2. Diagnose run-over-run — diff shuffle write bytes and input bytes, compare output row counts per stage, and check whether the partition predicate still prunes. (from runbook-spark-job-suddenly-costs-4x )
3. Rank the four usual suspects — a join fanning out, a dropped partition filter re-reading full history, a small-file explosion after a partitioning change, autoscaling widened because a stage became slow. (from runbook-spark-job-suddenly-costs-4x )
4. Read the masking trap — autoscaling hides the cause because a slow stage widens the cluster and buys speed while the fan-out stays; compare run-over-run, not day-over-day, or traffic deltas will fool the diff. (from runbook-spark-job-suddenly-costs-4x )
5. Fix per cause — dedupe the many-side on the join key, restore the partition predicate and re-run, or OPTIMIZE/compact and re-run. (from runbook-spark-job-suddenly-costs-4x )
6. Make the alarm precise — $ per pipeline run is a stable denominator, and cost-per-unit spikes the day a join starts fanning out. (from skill-cost-unit-economics )
7. Prevent the repeat — a cost-per-run unit metric with a 25% regression alarm and a CI guard on shuffle/scan metrics (hook-spark-cost-check). (from runbook-spark-job-suddenly-costs-4x )

Check yourself

• The run-over-run diff of shuffle write bytes identifies the guilty stage, and the fix brings cost-per-run back to baseline.
• A cost-per-run alarm with a regression threshold exists, so the next spike pages within one run instead of one month.
• Warehouses touched by agents sit behind a resource monitor with a hard credit cap.

Worked example

# from the event log
grep -E 'Shuffle Write|Input Size' spark.log | tail -40
SELECT count(*) FROM stage_output_pre JOIN ... -- compare to source count

$ per run, shuffle write bytes, input bytes, output rows

Use a dedicated service user with a least-privilege role, a resource monitor
with a hard credit cap, and a read-only role by default.

Sourced from

• Spark job suddenly costs 4x
• Cost Unit-Economics Reporter
• Snowflake MCP Server (Labs)

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Autoscaling masks the cause — a slow stage widens the cluster and buys speed while the fan-out stays.
• Compare run-over-run, not day-over-day — traffic deltas will fool the diff.
• A dropped partition filter reads full history — the join looks right, the input bytes don't.

Production considerations

• Monitor $ per run, shuffle write bytes, input bytes, and output rows together.
• Add a CI guard on shuffle/scan metrics (hook-spark-cost-check) so cost regressions fail before deploy.

Diagram

cost spike incident
   |
   |- invoice alert / $-per-run spike --> name the incident
   |
   |- diff run-over-run: shuffle write bytes, input bytes
   |
   |- suspects: fan-out | dropped partition filter | small files | autoscaling loop
   |
   |- fix per cause: dedupe many-side | restore predicate | OPTIMIZE/compact
   |
   |- resource monitor with hard credit cap --> blast radius bounded
   |
    --> revert the suspect deploy first; diagnose on the reverted baseline

Registry stats