DE//AGENT
Explore
Agent community advanced

Spark Optimizer

Diagnose and fix slow or expensive Spark jobs from their metrics.

At a glance

What Diagnose and fix slow or expensive Spark jobs from their metrics.
Where Spark / PySpark

Purpose

Diagnose and fix slow or expensive Spark jobs from their metrics.

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Registry stats

Verification community · Sep 28, 2026
Completeness 80
Trust multiplier ×1
Composite 34.5 · adequate
Freshness 26 days since last source check
Difficulty advanced
Version 0.2.1
License Apache-2.0
Security flags 1

Workflow

  • Parse task-duration and shuffle metrics.
  • [object Object]
  • Propose the single highest-impact change with expected effect.
  • Verify against the baseline metrics.

Inputs

  • Spark event log or Spark UI metrics
  • The job source

Outputs

  • Root-cause classification
  • One recommended change
  • Before/after metric expectation

Guardrails

  • Never suggests repartition(1)
  • Requires metrics; will not guess without them

Example

Sees p99 task duration 34 min vs p50 40 s, confirms key skew, recommends AQE skew join plus salting the top key.

Evaluation

Root-cause classification matches a set of labelled Spark event logs.

Tools

  • read_file
  • skill-spark-skew-diagnosis

Skills

MCP Servers