Bruin Academy
Guide
End-to-End Pipeline: NYC Taxi
Build a complete data pipeline from scratch using real NYC taxi data - from ingestion to staging to reports, all orchestrated with Bruin and DuckDB.
Skip ahead
The whole project is one command. Run it, point your coding agent at the Bruin MCP, and it can configure and run everything itself.
$ bruin init zoomcampThree ways to go from here:
- Self-service it. Let the agent configure and run the whole thing, and ask it questions as you go.
- Follow this tutorial. Slower, and it explains why each setting matters - the part an agent will not guess for you.
- Hand the tutorial to the agent. Point it at this page and have it work through the steps with you.
What
Build a real data pipeline end-to-end using NYC taxi trip data. Go from raw API data to clean, aggregated reports - learning ingestion, transformation, quality checks, and AI-assisted development along the way.
- End-to-end ELT pipeline: Python ingestion, SQL staging, reporting layers with quality checks
- Full orchestration with dependency management, execution order, and visual lineage
- AI integration via
bruin ai enhanceand Bruin MCP
How
- Bruin CLI orchestrates the pipeline; DuckDB serves as the local data warehouse
- Python assets ingest from the NYC TLC API; SQL assets handle transformations
- Bruin MCP connects an AI agent for pipeline development and data analysis
Before you start
- Bruin CLI installed
- VS Code or Cursor with the Bruin extension
- Familiarity with Bruin Core Concepts (recommended)
Guide overview
- 1Install BruinInstall the CLI, VS Code extension, and MCP so local pipeline development is ready.
- 2Build Your First PipelineCreate and run a simple DuckDB pipeline using the Chess.com public API.
- 3Core ConceptsUnderstand projects, pipelines, assets, variables, checks, and commands before larger builds.
- 4End-to-End Pipeline: NYC TaxiBuild a complete pipeline from ingestion to staging, reports, checks, and AI-assisted iteration.
Resources
Get help & contribute