Hook community advanced
Spark Cost Check
Fail a build when a Spark job's shuffle/spill/scan metrics regress past a threshold.
At a glance
| What | Fail a build when a Spark job's cost metrics regress past a threshold. |
|---|---|
| Where | Cost Engineering |
Purpose
Fail a build when a Spark job's cost metrics regress past a threshold.
Our assessment
Limitations
- Needs prior runs to build its 5-run baseline, so a cold start has nothing to compare against.
Security scan
Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.
Implementation
After the job's test run, read shuffle/spill/scan metrics from the event log, compare against the stored baseline (median of last 5), and fail if any metric grew by more than 25%. Emit the offending stage and metric.
Registry stats
| Verification | community · Sep 24, 2026 |
|---|---|
| Completeness | 74 |
| Trust multiplier | ×0.75 |
| Composite | 31.1 · adequate |
| Freshness | 30 days since last source check |
| Difficulty | advanced |
| Version | 0.1.3 |
| License | Apache-2.0 |
| Security flags | 2 |
Example
Shuffle write grows 3.9x after a join change → build fails with the stage id.
Did this solve your problem?
Spark Cost Check