Review a PySpark transform for silent quality failures

At a glance

Learning objectives

• Identify the five classes of silent failure in a Spark transform
• Add pre/post row-count and null assertions to a job
• Report findings as severity, file:line, rule, patch

What you need

• A PySpark transform about to be merged
• The intended grain of the output table
• [object Object]

Steps

1. Run this when a PySpark transform is about to be merged, or when a downstream table's row count changed without a code change. (from skill-spark-data-quality-review )
2. Null propagation — every outer join introduces nullable right-side keys; confirm downstream code coalesces or filters them. (from skill-spark-data-quality-review )
3. Join fan-out — assert key uniqueness on the intended 1-side before joining. (from skill-spark-data-quality-review )
4. Silent coercion — `cast` failures return null, not errors; prefer `try_cast` and count the nulls it produces. (from skill-spark-data-quality-review )
5. Non-determinism — `dropDuplicates` without a sort key keeps an arbitrary row. (from skill-spark-data-quality-review )
6. Missing assertions — every write should be followed by a row-count and a null-rate check on the primary key. (from skill-spark-data-quality-review )

Check yourself

• Every write in the transform is followed by a row-count and a null-rate check on the primary key.
• Findings come back as `severity | file:line | rule | suggested patch`.

Worked example

1. Null propagation — every outer join introduces nullable right-side keys.
   Confirm downstream code coalesces or filters them.
2. Join fan-out — assert key uniqueness on the intended 1-side before joining.
3. Silent coercion — `cast` failures return null, not errors. Prefer `try_cast`
   and count the nulls it produces.
4. Non-determinism — `dropDuplicates` without a sort key keeps an arbitrary row.
5. Missing assertions — every write should be followed by a row-count and a
   null-rate check on the primary key.

Run this when a PySpark transform is about to be merged, or when a downstream
table's row count changed without a code change.

Output: a findings list — severity | file:line | rule | suggested patch

Sourced from

• Spark Data Quality Review

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• `cast` failures return null, not errors — count the nulls try_cast produces.
• `dropDuplicates` without a sort key keeps an arbitrary row.
• Every outer join introduces nullable right-side keys — confirm downstream code coalesces or filters them.

Production considerations

• Every write should be followed by a row-count and a null-rate check on the primary key.
• Report findings as `severity | file:line | rule | suggested patch` so they are actionable in a PR.

Diagram

transform review
   |
   |- null propagation --> outer joins --> coalesce or filter downstream
   |
   |- join fan-out --> assert 1-side key uniqueness
   |
   |- silent coercion --> try_cast + count nulls
   |
   |- non-determinism --> sort key before dropDuplicates
   |
   |- missing assertions --> row-count + null-rate after every write

Registry stats