Start a PySpark project from the template

At a glance

Learning objectives

• Scaffold a PySpark project with the template
• Arrange transforms and offline test fixtures
• Add pre/post row-count and null assertions to a job

What you need

• An agent host with template support
• Docker for the containerised test run
• [object Object]

Steps

1. Scaffold the project: `amp template new de-agent/pyspark-project ./my-pipeline`. (from tpl-pyspark-project )
2. Check the scaffold — src/transforms, tests with a local Spark session, a Dockerfile, and a Makefile that runs lint + tests offline. (from tpl-pyspark-project )
3. Put transforms in src/transforms/ (orders_daily.py), readers in src/io/, and 3-10 row fixtures in tests/data/. (from tpl-pyspark-project )
4. Run the Makefile's lint + tests offline — no cluster needed. (from tpl-pyspark-project )
5. Add pre/post row-count and null assertions to the first job. (from skill-spark-data-quality-review )

Check yourself

• The Makefile runs lint + tests offline against the 3-10 row fixtures in tests/data/.
• The first transform ships with a row-count and a null-rate check on its primary key.

Worked example

amp template new de-agent/pyspark-project ./my-pipeline

pyspark-project/
  pyproject.toml
  Makefile
  Dockerfile
  src/
    transforms/
      orders_daily.py
    io/readers.py
  tests/
    data/            # 3-10 row fixtures
    test_orders_daily.py

Missing assertions — every write should be followed by a row-count and a
null-rate check on the primary key.

Sourced from

• PySpark Project Template
• Spark Data Quality Review

Security scan

Last sweep 2026-10-24: 0 finding(s), 0 marked as examples.

Common gotchas

• tests/data/ is for 3-10 row fixtures — a fixture that needs a cluster defeats the offline Makefile.
• Transforms live in src/transforms/ and I/O in src/io/ — mixing them breaks the layout the tests expect.
• The Makefile runs lint + tests offline — a transform that only runs on a cluster has no offline test.

Production considerations

• Keep tests with a local Spark session so CI needs no cluster.
• Pair the template with the Spark data-quality review skill before merging transforms.

Diagram

amp template new
   |
   |- scaffold --> src/transforms + src/io + tests/data (3-10 rows)
   |
   |- make lint + tests --> offline, local Spark session
   |
   |- first transform --> row-count + null assertions --> ready for review

Registry stats