DE//AGENT
Explore
Runbook community intermediate severity: high

Missing partition

An expected date partition is absent, so a downstream report silently excludes a day.

At a glance

When A date partition is absent and reports quietly exclude a day; Freshness checks pass while completeness fails
Where Batch Processing
Why A missing partition is silent data loss in every downstream report, and the window to catch it cleanly shrinks every day.

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Symptoms

  1. A date partition is absent from the table
  2. A downstream report silently excludes a day
  3. Freshness check passes but completeness fails

Incident notebook

Impact

Every downstream report silently excludes a day; decisions made on incomplete history look authoritative.

Escalation

Page the on-call if the partition feeds revenue or executive reporting; inform report consumers the same day either way.

Evidence to collect first

  • Partition listing diffed against the expected calendar
  • The job run's logical date, UTC and local timestamps
  • Source-side proof the data existed (or didn't) for that date

Rollback

Not applicable — recover forward: re-run the logical date once the timezone mapping is confirmed; never delete-and-hope.

Follow-up

  • Add the completeness assertion: every expected partition exists
  • Store the logical date as a partition column, not read-time derived

Diagnosis

  • List partitions for the range and diff against the calendar.
  • Check the job's logical date vs UTC vs local conversion.
  • Confirm whether the source actually had data.

Possible causes

  • Upstream produced no file for that date (source gap)
  • The job wrote to the wrong date due to timezone handling
  • A partially failed run left an empty or absent partition

Commands

aws s3 ls s3://bucket/table/ --recursive | grep 'dt=2026-09'
SELECT dt, count(*) FROM tbl GROUP BY dt ORDER BY dt DESC LIMIT 30;

Fix

  • Re-run the specific logical date once the source is confirmed.
  • Backfill the gap partition with the corrected timezone mapping.

Prevention

  • A completeness check that asserts every expected partition exists.
  • Store the logical date explicitly as a partition column, not derived at read time.

Monitoring

  • partition count per day, completeness assertions

Common gotchas

  • Confirm the source actually had data before backfilling — otherwise you bake the gap in.
  • Timezones: job logical date vs UTC vs local — diff all three before re-running.
  • An empty partition and an absent partition both read as 'no data' downstream.

Production considerations

  • Assert every expected partition exists (completeness check, not freshness).
  • Store the logical date as a partition column; never derive it at read time.

Tradeoffs

Choose this when Costs & caveats
partition-level completeness makes gaps unmissable strict partition alerts can get noisy during backfill seasons

Diagram

partition missing
   |
   |- source gap --> confirm upstream had data --> re-run the logical date
   |
   |- timezone mapping wrong --> fix mapping --> backfill corrected date

Registry stats

Verification community · Sep 16, 2026
Completeness 75
Trust multiplier ×0.5
Composite 29.25 · adequate
Freshness 38 days since last source check
Difficulty intermediate
Severity high
Version 1.0.0
License CC-BY-4.0
Security flags 3