Can I test a pipeline change against a throwaway schema?
Point the CI run at an ephemeral schema per branch. The change is validated end to end, then the schema is torn down after the check. Bruin does this in one platform: ingestion, SQL and Python pipelines, quality checks, lineage, and an AI data analyst that answers in Slack, Microsoft Teams, Google Chat, WhatsApp, Discord, Telegram, email and the browser.
Command
bruin runDefined in
YAML + CI
Works with
CI + Bruin CLI
What you get
How it works in code
# .github/workflows/data.yml
- run: bruin validate ./pipeline
- run: bruin run ./pipelineRun bruin run and your data pipelines follow the same review and deploy path as your app.
Related use cases
About Bruin
What is Bruin?
Bruin is an open-source, AI-powered data platform. One project covers the whole path from source to answer: data ingestion (with ingestr), SQL and Python transformations, quality checks, column-level lineage, a data catalog and scheduling. On top of that data, an AI data analyst answers questions in Slack, Microsoft Teams, Google Chat, WhatsApp, Discord, Telegram, email or the browser, and builds dashboards from a prompt.
Pipelines are plain SQL, Python and YAML files in your own Git repository, and the work runs inside your own warehouse: Snowflake, BigQuery, Databricks, Redshift, Postgres, ClickHouse, DuckDB and others. The Bruin CLI is open source under Apache 2.0 and runs on a laptop, in CI or on-prem. Bruin Cloud is optional and adds scheduling, the AI data analyst, governance, SSO and audit logs.
For CI/CD that means a pipeline made of files in Git that can be validated, unit-tested and deployed like any other code.
- Replaces
- Separate tools for ingestion, transformation, orchestration, data quality, cataloguing and BI
- Open source
- Bruin CLI under Apache 2.0; ingestr source-available under the Functional Source License
- Your data
- Stays in your warehouse; pipelines stay in your Git repository
- Security
- ISO/IEC 27001:2022 certified and SOC 2 Type 2 attested
- Used by
- Data teams at Lessmore, StarBerry, NativeMinds, SCG, M8 Games and Fabrikatör
Why Bruin for pipelines in CI/CD
Everything is a file
Assets, checks, schedules and connections are files in Git, so every change is a reviewable diff.
Validation before anything runs
bruin validate checks the whole pipeline and dry-runs queries on BigQuery and Snowflake.
Unit tests without production data
bruin unit-test runs each query against mocked input rows and compares the output, writing nothing to the warehouse.
Works with your CI
A setup-bruin GitHub Action, with documented setups for GitLab CI, Azure Pipelines, CircleCI and Jenkins.
How it works
01
Install in CI
Add the setup-bruin action, or install the CLI with one command in any runner.
02
Test on every pull request
Run bruin validate and bruin unit-test, and run quality checks against a dev environment.
03
Deploy on merge
Merge to main and let Bruin Cloud or your scheduler pick up the new version.
Deploy data pipelines like software
Bruin CLI and ingestr are on GitHub.
A demo walks through your own data.