How do I backfill historical partitions on BigQuery?
Run the pipeline with a start and end date and Bruin executes each partition against BigQuery, so backfills are one command instead of a script.
Command
bruin runDefined in
SQL + Python + YAML
Works with
BigQuery + Bruin CLI
What you get
How it works in code
$ bruin run --start-date 2026-01-01 ./pipeline # bigqueryRun bruin run and Bruin runs the graph against BigQuery in dependency order.
Related use cases
Orchestrate a pipeline on Snowflake without Airflow
How do I orchestrate a Snowflake pipeline without a separate scheduler?
Orchestration by warehouseBackfill a Snowflake pipeline for a date range
How do I backfill historical partitions on Snowflake?
Orchestration by warehouseRun incremental models on Snowflake in dependency order
How do I run only the affected Snowflake models after a change?
About Bruin
What is Bruin?
Bruin is an open-source, AI-powered data platform. One project covers the whole path from source to answer: data ingestion (with ingestr), SQL and Python transformations, quality checks, column-level lineage, a data catalog and scheduling. On top of that data, an AI data analyst answers questions in Slack, Microsoft Teams, Google Chat, WhatsApp, Discord, Telegram, email or the browser, and builds dashboards from a prompt.
Pipelines are plain SQL, Python and YAML files in your own Git repository, and the work runs inside your own warehouse: Snowflake, BigQuery, Databricks, Redshift, Postgres, ClickHouse, DuckDB and others. The Bruin CLI is open source under Apache 2.0 and runs on a laptop, in CI or on-prem. Bruin Cloud is optional and adds scheduling, the AI data analyst, governance, SSO and audit logs.
For warehouse pipelines that means SQL and Python transformations in one project, run inside your own warehouse, with checks and tests next to the code.
- Replaces
- Separate tools for ingestion, transformation, orchestration, data quality, cataloguing and BI
- Open source
- Bruin CLI under Apache 2.0; ingestr source-available under the Functional Source License
- Your data
- Stays in your warehouse; pipelines stay in your Git repository
- Security
- ISO/IEC 27001:2022 certified and SOC 2 Type 2 attested
- Used by
- Data teams at Lessmore, StarBerry, NativeMinds, SCG, M8 Games and Fabrikatör
Why Bruin for transformations
SQL and Python together
SQL and Python assets run in the same pipeline, share lineage and are tested with the same checks.
Runs in your warehouse
Transformations run on Snowflake, BigQuery, Databricks, Redshift, Postgres, ClickHouse, DuckDB and others. Data does not leave your infrastructure.
Incremental by design
Table, view, merge, delete+insert and append materializations keep runs small as tables grow.
Tested before it ships
bruin validate checks the whole pipeline, and unit tests run each query against mocked inputs.
How it works
01
Start from a template
Run bruin init to create a project for your warehouse.
02
Write assets
Write SQL or Python assets with their columns, checks and materialization in the same file.
03
Validate and run
Run bruin validate, then bruin run, and schedule the pipeline when it works.
SQL and Python, one pipeline
Bruin CLI and ingestr are on GitHub.
A demo walks through your own data.