How do I orchestrate ingestion and transformation together?
Put an ingestr ingestion step and your SQL and Python transforms in the same Bruin pipeline. Bruin runs load then model in dependency order under one command, so ingestion and transformation stay in sync.
Command
bruin runDefined in
YAML + SQL + Python
Works with
ingestr + Bruin CLI
What you get
Ingestion and transforms togetherdependency orderone runtime
How it works in code
/* @bruin
name: mart.orders
depends: [ingest.orders]
materialization:
type: table
@bruin */
SELECT * FROM ingest.ordersRun bruin run and Bruin runs ingestion before the transforms that depend on it.
Related use cases
Orchestration and scheduling
Orchestrate a pipeline without Airflow
Can I orchestrate SQL and Python steps without standing up Airflow?
Orchestration and schedulingSchedule a pipeline to run daily
How do I schedule a Bruin pipeline to run on a cadence?
Orchestration and schedulingDefine dependencies between assets
How do I declare that one model depends on another?
SQL and Python, one pipeline
Open source. Define assets, declare dependencies, run the graph with bruin run.