How do I run and validate Redshift pipelines from GitLab CI?
Call bruin validate and bruin run in a GitLab CI job against Redshift. Connections come from CI secrets, and a failing check blocks the merge before bad data lands in Redshift. Bruin does this in one platform: ingestion, SQL and Python pipelines, quality checks, lineage, and an AI data analyst that answers in Slack, Microsoft Teams, Google Chat, WhatsApp, Discord, Telegram, email and the browser.
Command
bruin runDefined in
YAML + CI
Works with
GitLab CI + Redshift
What you get
How it works in code
# gitlab job
- run: bruin validate ./pipeline
- run: bruin run ./pipeline # redshiftRun bruin run and the Redshift pipeline runs on merge or on schedule from GitLab CI.
Related use cases
Run Snowflake pipelines in GitLab CI
How do I run and validate Snowflake pipelines from GitLab CI?
Run pipelines in GitLab CIRun BigQuery pipelines in GitLab CI
How do I run and validate BigQuery pipelines from GitLab CI?
Run pipelines in GitLab CIRun Postgres pipelines in GitLab CI
How do I run and validate Postgres pipelines from GitLab CI?
About Bruin
What is Bruin?
Bruin is an open-source, AI-powered data platform. One project covers the whole path from source to answer: data ingestion (with ingestr), SQL and Python transformations, quality checks, column-level lineage, a data catalog and scheduling. On top of that data, an AI data analyst answers questions in Slack, Microsoft Teams, Google Chat, WhatsApp, Discord, Telegram, email or the browser, and builds dashboards from a prompt.
Pipelines are plain SQL, Python and YAML files in your own Git repository, and the work runs inside your own warehouse: Snowflake, BigQuery, Databricks, Redshift, Postgres, ClickHouse, DuckDB and others. The Bruin CLI is open source under Apache 2.0 and runs on a laptop, in CI or on-prem. Bruin Cloud is optional and adds scheduling, the AI data analyst, governance, SSO and audit logs.
For CI/CD that means a pipeline made of files in Git that can be validated, unit-tested and deployed like any other code.
- Replaces
- Separate tools for ingestion, transformation, orchestration, data quality, cataloguing and BI
- Open source
- Bruin CLI under Apache 2.0; ingestr source-available under the Functional Source License
- Your data
- Stays in your warehouse; pipelines stay in your Git repository
- Security
- ISO/IEC 27001:2022 certified and SOC 2 Type 2 attested
- Used by
- Data teams at Lessmore, StarBerry, NativeMinds, SCG, M8 Games and Fabrikatör
Why Bruin for pipelines in CI/CD
Everything is a file
Assets, checks, schedules and connections are files in Git, so every change is a reviewable diff.
Validation before anything runs
bruin validate checks the whole pipeline and dry-runs queries on BigQuery and Snowflake.
Unit tests without production data
bruin unit-test runs each query against mocked input rows and compares the output, writing nothing to the warehouse.
Works with your CI
A setup-bruin GitHub Action, with documented setups for GitLab CI, Azure Pipelines, CircleCI and Jenkins.
How it works
01
Install in CI
Add the setup-bruin action, or install the CLI with one command in any runner.
02
Test on every pull request
Run bruin validate and bruin unit-test, and run quality checks against a dev environment.
03
Deploy on merge
Merge to main and let Bruin Cloud or your scheduler pick up the new version.
Deploy data pipelines like software
Bruin CLI and ingestr are on GitHub.
A demo walks through your own data.