Migrations/Migrate from AirflowData Engineer

How do I move my Airflow workflow to Bruin on Postgres?

Point Bruin at your Postgres connection and bring your Airflow logic across as Bruin assets. Ingestion, transformation, quality, and lineage then run against Postgres from one CLI.

Command

bruin run

Defined in

SQL + Python + YAML

Works with

Airflow + Postgres + Bruin

What you get

Consolidated stackwarehouse-nativeCLI-first

How it works in code

$ bruin run ./pipeline  # against postgres

Run bruin run and your whole workflow runs against Postgres from one tool.

About Bruin

What is Bruin?

Bruin is an open-source, AI-powered data platform. One project covers the whole path from source to answer: data ingestion (with ingestr), SQL and Python transformations, quality checks, column-level lineage, a data catalog and scheduling. On top of that data, an AI data analyst answers questions in Slack, Microsoft Teams, Google Chat, WhatsApp, Discord, Telegram, email or the browser, and builds dashboards from a prompt.

Pipelines are plain SQL, Python and YAML files in your own Git repository, and the work runs inside your own warehouse: Snowflake, BigQuery, Databricks, Redshift, Postgres, ClickHouse, DuckDB and others. The Bruin CLI is open source under Apache 2.0 and runs on a laptop, in CI or on-prem. Bruin Cloud is optional and adds scheduling, the AI data analyst, governance, SSO and audit logs.

For migrations that means moving one pipeline at a time, with the old and new versions running side by side until the numbers match.

Replaces
Separate tools for ingestion, transformation, orchestration, data quality, cataloguing and BI
Open source
Bruin CLI under Apache 2.0; ingestr source-available under the Functional Source License
Your data
Stays in your warehouse; pipelines stay in your Git repository
Security
ISO/IEC 27001:2022 certified and SOC 2 Type 2 attested
Used by
Data teams at Lessmore, StarBerry, NativeMinds, SCG, M8 Games and Fabrikatör

Why Bruin for migrations

  • Incremental, not a rewrite

    Move one pipeline, compare its output, then move the next. Nothing has to switch over in one go.

  • Start from what you have

    bruin import database turns existing warehouse tables into assets, and dbt models port with their Jinja intact.

  • Run inside your current scheduler

    Call Bruin from an existing Airflow DAG while you move, then retire the DAG later.

  • Prove the numbers match

    bruin data-diff compares the old and new tables before you switch.

How it works

  1. 01

    Import

    Map existing tables with bruin import database and start a project.

  2. 02

    Port one pipeline

    Move one ingestion and its models, add checks, and run both versions side by side.

  3. 03

    Compare and switch

    Diff the outputs, switch consumers to the Bruin tables, and turn off the old job.

Migrate incrementally, not all at once

Bruin CLI and ingestr are on GitHub.

A demo walks through your own data.

Sign up to our newsletter

Practical updates on open-source data pipelines, AI analysts, governance, and what we are shipping at Bruin.

The signup form is hosted by Brevo. Allow marketing cookies to load it.