DE//AGENT
Explore
Skill community intermediate

Pipeline Test Author

Write integration tests for a pipeline that run in seconds, offline.

Builds a local DuckDB/pyarrow harness with tiny fixtures so pipeline logic is testable in CI without a cluster, and asserts on invariants, not snapshots.

npx de-agent-platform add pipeline-test-author

At a glance

What Write integration tests for a pipeline that run in seconds, offline.
Where Testing · tech-python · tech-duckdb · tech-parquet

Learning objectives

  • Test a transform without a Spark cluster
  • Assert invariants instead of golden files
  • Make a pipeline test deterministic

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Details

Pipeline Test Author

The harness

Keep fixtures as 3-10 row Parquet/CSV files under tests/data/. Load them with DuckDB or pyarrow. No cluster, no network, < 2s.

Assert invariants, not snapshots

  • count(*) is stable across a re-run (idempotency)
  • no nulls in the grain key
  • sum(revenue) equals the source sum within tolerance
  • a quarantined row never appears in the published table

Rule

If a test needs the network or a running warehouse, it is not a unit test.

Registry stats

Verification community · Sep 27, 2026
Completeness 100
Trust multiplier ×0.75
Composite 38 · adequate
Freshness 27 days since last source check
Difficulty intermediate
Version 0.2.0
License Apache-2.0
Security flags 2