Install cost guardrails before the invoice

At a glance

Learning objectives

• Define a unit metric per pipeline
• Baseline the last N runs and fail CI past a threshold
• Add hard caps on top of CI guards

What you need

• Cost-relevant metrics per run (shuffle/spill/scan)
• CI where a build can fail on a metric check
• [object Object]

Steps

1. Define a unit metric per pipeline ($ per run, per 1M rows). (from pattern-cost-guardrails )
2. Baseline the last N runs. (from pattern-cost-guardrails )
3. Fail CI on regression > threshold. (from pattern-cost-guardrails )
4. Add hard caps (warehouse resource monitors, quota limits). (from pattern-cost-guardrails )
5. Alert on the unit metric, never on total spend, so healthy growth doesn't page anyone and a regression does. (from pattern-cost-guardrails )
6. Verify the guard — shuffle write grows 3.9x after a join change → the build fails with the stage id. (from hook-spark-cost-check )

Check yourself

• A change that grows any metric > 25% vs the baseline fails CI with the offending stage and metric named.
• Healthy growth (more data, same cost per run) does not page anyone.

Worked example

Instrument, attribute, then guard.
  1. unit metric per pipeline ($ per run, per 1M rows)
  2. baseline the last N runs
  3. fail CI on regression > threshold
  4. hard caps (warehouse resource monitors, quota limits)
Alert on the *unit* metric, never on total spend, so healthy growth doesn't page
anyone and a regression does.

shuffle read/write bytes, spill (disk) bytes, task count, max task duration,
input bytes. These predict cost far better than wall-clock time.

After the job's test run, read shuffle/spill/scan metrics from the event log,
compare against the stored baseline (median of last 5), and fail if any metric
grew by more than 25%. Emit the offending stage and metric.

Example: shuffle write grows 3.9x after a join change → build fails with the stage id.

Sourced from

• Cost Guardrails
• Spark Cost Check

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• Spend varies with data growth and regressions hide in growth — total-spend dashboards cannot see them.
• Guardrails need metrics plumbing and a baseline to be maintained — skip the baseline and the guard has nothing to compare.
• Needs prior runs to build its 5-run baseline, so a cold start has nothing to compare against.

Production considerations

• Keep the baseline as the median of the last 5 runs.
• Gate deploys on shuffle/scan metrics, and keep resource monitors as the hard cap.

Diagram

run
   |
   |- metrics --> unit metric ($ per run / per 1M rows)
   |
   |- baseline last N runs --> within baseline --> pass
   |                          > regressed --> fail CI / alert
   |
   |- hard caps --> resource monitors + quotas --> invoice backstop

Registry stats