Hold the observability mental model: infrastructure metrics versus data signals

At a glance

Learning objectives

• Separate infrastructure metrics from data signals — freshness, volume, schema, distribution
• Place lineage and the catalog on the observability map, next to metrics and traces
• State what each layer answers and which tool owns it

What you need

• No cluster required — this is a concepts lesson
• The Prometheus and Grafana documentation open beside you

Steps

1. Ground the infra layer — the pull-based metrics model, PromQL, exporters, service discovery and alerting rules, with infra- and job-level metrics that sit underneath data-quality signals like freshness and volume. (from ref-prometheus-docs )
2. Ground the data layer's surface — dashboards, panel queries, alerting including the unified alerting model; the standard surface where pipeline freshness, volume and job-health dashboards live. (from ref-grafana-docs )
3. Add the instrumentation layer — traces, metrics and logs through a vendor-neutral API/SDK, the collector, and semantic conventions; increasingly the default for distributed pipelines and services. (from ref-otel-docs )
4. Add the lineage layer — an open specification for lineage events with integrations for Airflow, Spark and dbt; not a tool or a service, a standard that makes cross-tool lineage possible. (from ref-openlineage-docs )
5. Add the catalog layer — the open metadata model for ownership and lineage, ingestion framework, search, and governance integration; the vocabulary for catalog-related pipeline design. (from ref-datahub-docs )
6. See why lineage earns its layer — technical lineage (tables and jobs) versus business lineage (metrics and reports); a breaking change needs both. (from skill-lineage-explainer )
7. See that data signals can gate changes — a hook that warns when a change would silently widen the freshness SLA of a source. (from hook-freshness-pre-merge )

Check yourself

• You can name the four data signals and give one example threshold for each.
• Asked what lineage adds that metrics cannot, you answer — the blast radius and the owner.

Worked example

infra metrics   ─▶ is the job running?   (pull-based metrics model, PromQL, alerting rules)
data signals    ─▶ is the data usable?   (freshness, volume, schema, distribution)
traces          ─▶ what ran, how long?   (vendor-neutral API/SDK + collector)
lineage events  ─▶ what feeds what?      (open spec; cross-tool lineage)
catalog         ─▶ who owns it?          (metadata model; ownership + lineage)

Technical lineage — which tables/jobs depend on this column.
Business lineage — which metrics/reports depend on it.
A breaking change needs both, because a CFO does not read table names.

Sourced from

• Prometheus Documentation
• Grafana Documentation
• OpenTelemetry Documentation
• OpenLineage Documentation
• DataHub Documentation
• Lineage Explainer
• Freshness Pre-Merge Check

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• A green job is not fresh data — the schedule can succeed forever while the source quietly stops changing.
• Infrastructure metrics sit underneath data signals — do not stop at the infra layer and call it data observability.
• Lineage is not decoration — without it a freshness alert cannot name a blast radius.

Production considerations

• Instrument in layers — infra metrics from the scheduler, data signals from the tables, lineage events from the jobs.
• Keep ownership in the catalog, not in a runbook — alerts resolve faster when the owner is one query away.

Diagram

observability layers
   |
   |- infra: CPU, lag, job success --> Prometheus pull model + PromQL
   |
   |- data signals: freshness | volume | schema | distribution
   |
   |- instrumentation: traces + metrics + logs (vendor-neutral API/SDK)
   |
   |- lineage: what feeds what --> blast radius + earliest SLA
   |
    --> catalog: who owns it, is it safe to use

Registry stats