Read a Spark event log with the Spark Lens plugin

At a glance

Learning objectives

• Turn a Spark event log into a task histogram
• Flag skew and shuffle amplification with /spark-analyze
• Store a cost baseline with /spark-baseline

What you need

• A Spark event log or Spark UI metrics
• The job source
• [object Object]

Steps

1. Install the plugin: `amp plugin add de-agent/spark-lens`. (from plugin-spark-lens )
2. Point it at a Spark event log — it parses the log into a task histogram and flags skew and shuffle amplification. (from plugin-spark-lens )
3. Run `/spark-analyze` on the event log. (from plugin-spark-lens )
4. Run `/spark-explain-plan` before shipping the change. (from plugin-spark-lens )
5. Run `/spark-baseline` to store the cost baseline. (from plugin-spark-lens )
6. Wire the guard — shuffle write grows 3.9x after a join change → build fails with the stage id. (from hook-spark-cost-check )

Check yourself

• `/spark-analyze` produces a task histogram from the event log and names skew or shuffle amplification when present.
• A join change that grows shuffle write 3.9x fails the build with the stage id.

Worked example

amp plugin add de-agent/spark-lens

/spark-analyze        # task histogram; flags skew and shuffle amplification
/spark-explain-plan   # check the join strategy before shipping the change
/spark-baseline       # store the cost baseline the CI guard enforces

After the job's test run, read shuffle/spill/scan metrics from the event log,
compare against the stored baseline (median of last 5), and fail if any metric
grew by more than 25%. Emit the offending stage and metric.

Example: shuffle write grows 3.9x after a join change → build fails with the stage id.

Sourced from

• Spark Lens
• Spark Cost Check
• Spark job suddenly costs 4x

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• The cost guard needs prior runs to build its 5-run baseline — a cold start has nothing to compare against.
• The plugin's permissions are filesystem read-only with no network — it analyzes logs you provide, it does not fetch metrics.
• Event-log retention matters — rotated logs close the run-over-run comparison window.

Production considerations

• Store a baseline per job and compare against the median of the last 5 runs.
• The plugin ships the skew-diagnosis and cost-regression-guard skills — use them for the root-cause step.

Diagram

event log
   |
   |- /spark-analyze --> task histogram --> skew / shuffle amplification flags
   |
   |- /spark-explain-plan --> broadcast vs sort-merge
   |
   |- /spark-baseline --> stored baseline --> CI guard fails > 25% growth

Registry stats