Spark / PySpark

Start a PySpark project from the template

Scaffold a testable PySpark project with one command and learn where each piece lives: transforms in src/, 3-10 row fixtures in tests/data/, and a Makefile that runs lint plus tests offline.

Aggregations and windows without the skew

• Read p50 vs p99 task durations to confirm an aggregation is skewed
• Salt a hot key and aggregate twice

Diagnose a skewed Spark stage

• Read task-level duration histograms to confirm skew
• Choose between AQE skew join, salting and broadcast

Read a Spark event log with the Spark Lens plugin

Install the Spark Lens plugin and work its three commands over a Spark event log: /spark-analyze to build a task histogram and flag skew, /spark-explain-plan to check the join strategy, and /spark-baseline to store the cost baseline the guard enforces.

Review a PySpark transform for silent quality failures

Run the five-check pass over a PySpark transform before it merges: null propagation, join fan-out, silent coercion, non-determinism, and missing post-write assertions.

The 3am batch incident path

• Collect the four pieces of evidence a failing Spark run needs
• Decide between revert, a config change, and a code fix

Tune a slow job — memory, partitions, baselines

Run the tuning loop on a slow batch job: parse task-duration and shuffle metrics, classify the stage as skew, partition count, spill or scan, apply the single highest-impact change, and verify it against a stored baseline the CI guard enforces.

Spark Data Quality Review

• Identify the five classes of silent failure in a Spark transform
• Add pre/post row-count and null assertions to a job

Spark Skew Diagnosis

• Read task-level duration histograms to confirm skew
• Choose between AQE skew join, salting and broadcast

Spark Lens

Parses Spark event logs into a task histogram, flags skew and shuffle amplification, and compares against a stored cost baseline.

Spark Optimizer

Diagnose and fix slow or expensive Spark jobs from their metrics.

PySpark Project Template

A testable PySpark project: src/transforms, tests with a local Spark session, a Dockerfile, and a Makefile that runs lint + tests offline.

Spark job is slow

A batch Spark job's runtime has jumped — find whether it is skew, partition count, spill or over-scan.

Apache Spark Documentation

The authoritative reference for Spark SQL, the DataFrame API, tuning and the configuration catalogue, versioned per Spark release. Use it for exact semantics and config keys — when a job skews or slows, check the config reference and the tuning guide before changing code, alongside our skew-diagnosis and cost skills.