Run the Pipeline

You know how to write a query. Now make it safe to run every day.

• Turn a saved query into a pipeline with explicit dependencies, checks, tests, and evidence.
• Choose update strategies that tell when an order happened, when it reached the system, and which dates need repair.
• Give an agent a limited environment, rules it cannot bypass, and enough logs to investigate a failure.
• Ship a governed retail pipeline whose outputs can be reviewed, repaired, and rerun without trusting a confident answer.

How it works

Your coding agent teaches the course. It reads the project, teaches one lesson at a time, asks the quiz questions, and reviews the task after you finish it. These pages are a companion you can use alongside that conversation.

1. Paste this setup prompt into your agent in an empty folder.

The default environment is local DuckDB and needs no token. MotherDuck is optional. No lesson requires the cloud path.

2. Say next lesson. The agent teaches the next incomplete lesson, asks its questions one at a time, and gives you the task.

3. Do the task and say review my work. The agent reads the artifact or answer, runs the required checks, and grades it against the lesson rubric.

What you will build

The template is a repeatable retail pipeline with generated input data, staging tables, shared models, and reporting tables. Bruin calls the last group marts. The project is intentionally not clean: validation passes, while the first run fails only at mart.churn_risk so you can investigate a real defect in an earlier table later.

By the capstone, you will have a dependency graph, checks that stop bad output, unit tests, update rules that touch only the dates that changed, a customer history table that keeps old versions, late-data evidence, a dev run, a logging plan, and a written capstone defence. The target is a pipeline whose failures are visible, limited, and explainable.

The four phases

1. Build the pipeline - classify operational risk and read the dependency graph.
2. Make failure loud - turn audit questions into checks, tests, and controlled failure drills.
3. Run it repeatedly - make reruns safe, handle late data, compare filters, and use a separate dev database.
4. Operate with an agent - place guardrails at the right layer, investigate the shipped failure, read evidence, and ship the capstone.

Before you start

Take Design the Model first if you have not worked with model contracts, grain, joins, and data quality. This course assumes you can review SQL and focuses on the operational consequences of running it repeatedly.