skew

DE//AGENT currently lists 4 resources under skew: 3 tutorials, 1 skill. Everything below is derived from the live registry — counts, capabilities, verification state and security posture come from the curated records, never from marketing copy.

Documented learning objectives across these resources:

• Read task-level duration histograms to confirm skew — via Spark Skew Diagnosis
• Choose between AQE skew join, salting and broadcast — via Spark Skew Diagnosis
• Reproduce skew locally with a bounded dataset — via Spark Skew Diagnosis
• Read p50 vs p99 task durations to confirm an aggregation is skewed — via Aggregations and windows without the skew
• Salt a hot key and aggregate twice — via Aggregations and windows without the skew
• Right-size partitions when an oversized aggregate spills — via Aggregations and windows without the skew
• Attach the evidence a skew review requires — via Diagnose a skewed Spark stage
• Collect the four pieces of evidence a failing Spark run needs — via The 3am batch incident path

2 more objectives live on the individual resource pages.

These resources map to stage: Data Processing (4). Follow the four-phase roadmap to place this topic inside a progression instead of learning it in isolation.

Every entry, its shape and its trust state — sorted by the registry's composite score. This table is the honest version of a marketing comparison: no sponsored placements, no unverifiable claims.

How scores are computed: source quality score.

Highest composite score first:

1. Spark Skew Diagnosis score 38.5 — advanced
2. Aggregations and windows without the skew score 34.15 — intermediate
3. Diagnose a skewed Spark stage score 34.15 — intermediate

Scores rank provenance, completeness and trust — read how verification works before adopting anything.

Reach for these with production experience: Spark Skew Diagnosis, The 3am batch incident path.

Security-flagged entries: Spark Skew Diagnosis (2 flags). Flags mean the record declares capabilities like credential use or write access — read the security posture on each page before connecting.

• Spark Skew Diagnosis — Walks an agent through reading the Spark UI/event log to distinguish key skew, partition skew, and shuffle-write amplification, then selects between salting, AQE skew join, broadca… details
• Aggregations and windows without the skew — Aggregations and windows are shuffles in disguise. Treat the group-by key as a skew suspect, read p50 vs p99 and the spill bytes, then apply the matching fix — a salted two-stage a… details
• Diagnose a skewed Spark stage — One Spark stage hangs while other executors idle. Open the stage view, read the p50-vs-p99 task durations, and follow the skew skill's decision order: AQE skew join first, salting … details
• The 3am batch incident path — The nightly job is crawling or dead under memory pressure or skew. Follow the incident path: collect the four pieces of evidence, read p50 vs p99 and the spill bytes, decide revert… details

Real install commands from the registry — read the source before running any of them:

npx de-agent-platform add spark-skew-diagnosis

Spark Skew Diagnosis — review first

Adjacent topics: tutorial (3 shared), performance (2 shared), aqe (2 shared).

Study path: the roadmap places this topic in a progression · the scoring model explains the numbers · the trust rule catalog shows every security check these entries passed through.