DE//AGENT
Explore
Runbook community intermediate severity: high

Schema drift

The upstream schema changed without notice and a cast broke or a column silently nulled.

At a glance

When A column appeared, disappeared or changed type upstream without notice; A pipeline fails on a cast or silently nulls a column
Where Data Contracts
Why An uncontracted schema change is a loaded gun in the pipeline — hard casts break loudly, silent nulls break worse.

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Symptoms

  1. A column appears, disappears, or changes type upstream without notice
  2. A pipeline fails on a cast, or silently nulls a column
  3. Catalogue schema and file schema disagree

Incident notebook

Impact

Hard failures block pipelines; silent nulls poison columns while every check stays green — the second kind is the expensive one.

Escalation

Contact the producer team immediately with the schema diff; escalate if they cannot revert before the next consumer run.

Evidence to collect first

  • The schema diff: additive, type change, rename, or removal
  • The failing job's error, or the null-rate delta on affected columns
  • The contract version the producer was expected to honor

Rollback

Producer reverts or ships a compatibility shim; consumers may quarantine the affected columns in the interim.

Follow-up

  • Register the evolved contract; alert on any delta, even additive
  • Postmortem: how the change bypassed the contract process

Diagnosis

  • Diff the source schema against the last accepted contract version.
  • Classify the change: additive, type change, rename, removal.
  • Determine whether the failure is hard (job error) or silent (nulls).

Possible causes

  • Producer changed the schema with no contract or notice
  • A new column was added dynamically by the source
  • Type widened upstream (int → decimal) and broke a cast

Commands

diff <(parquet-tools schema old.parquet) <(parquet-tools schema new.parquet)
-- Glue/Athena: compare catalog columns to the Parquet footer

Fix

  • Additive change → evolve the schema forward (safe) and register it.
  • Type change → cast explicitly, or reject and quarantine.
  • Rename/removal → this is breaking; coordinate with consumers.

Prevention

  • A schema contract with a compatibility policy enforced in CI.
  • Alert on any schema delta, even additive ones.

Monitoring

  • schema-hash per source, drift events, quarantine counts

Common gotchas

  • Silent nulls are the dangerous case — the job stays green while the column dies.
  • Renames are removals in disguise; treat them as breaking.
  • Type widening upstream (int to decimal) breaks the narrow cast downstream.

Production considerations

  • Enforce a schema contract with a compatibility policy in CI.
  • Alert on any schema delta, even additive ones.

Tradeoffs

Choose this when Costs & caveats
an additive-first policy keeps most changes uneventful strict contracts push work onto producers, who may not have the time

Diagram

schema drift
   |
   |- additive --> evolve contract forward --> register
   |
   |- type change --> explicit cast, or reject + quarantine
   |
   |- rename / removal --> coordinate with consumers --> coordinated cutover

Registry stats

Verification community · Sep 26, 2026
Completeness 75
Trust multiplier ×0.75
Composite 34.75 · adequate
Freshness 28 days since last source check
Difficulty intermediate
Severity high
Version 1.2.0
License CC-BY-4.0
Security flags 2