Warehouse Pipelines/ReplicationData Engineer

How do I replicate a production database into Snowflake?

Bruin runs the whole pipeline against Snowflake from one project: ingestion, SQL and Python transformations, and quality checks are all assets in the same dependency graph, so bruin run resolves the order and executes them in sequence. Loads use an internal stage plus COPY INTO and incremental assets use MERGE on a primary key. The alternative is assembling an ingestion tool, a transformation framework, and an orchestrator, which works but leaves you owning the glue between them.

Command

bruin run

Defined in

SQL + Python + YAML

Works with

Snowflake + Bruin CLI

What you get

End-to-end pipelinestable materializationbuilt-in quality checks

How to do it

  1. 1

    Run bruin init and add a snowflake connection for Snowflake.

  2. 2

    Declare an ingestion asset so raw data lands in Snowflake via an internal stage plus COPY INTO.

  3. 3

    Add a SQL asset that models the raw table, with depends naming its upstream.

  4. 4

    Declare column checks on the keys and amounts that matter.

  5. 5

    Run bruin validate to confirm the graph resolves, then bruin run.

  6. 6

    Schedule it in CI or Bruin Cloud, and switch heavy assets to MERGE on a primary key.

How it works in code

/* @bruin
name: mart.orders
materialization:
  type: table
depends: [raw.orders]
columns:
  - name: order_id
    checks:
      - name: unique
@bruin */

SELECT order_id, customer_id, order_total
FROM raw.orders

Run bruin run and Bruin builds the asset on Snowflake and runs its checks before anything downstream reads it.

Worth knowing

On Snowflake, a virtual warehouse left running with no auto-suspend bills credits while doing nothing

Other ways to do this

Bruin is not always the right answer. Here is where the alternatives are stronger.

OptionWhen it is the better choice
BruinBuild and run a complete Snowflake pipeline with Bruin: ingest, transform in SQL or Python, and check the output, all from one project.
dbt + Fivetran + AirflowThe conventional split. Mature and well documented, but three tools to run and integrate for one Snowflake pipeline.
SQLMeshStrong on Snowflake with virtual environments that cut the compute cost of reviewing a change. Transformation only, so you still need ingestion.
Native Snowflake toolingStaying inside Snowflake avoids another vendor, at the cost of portability if you ever move warehouse.

Common questions

How do I build an end-to-end data pipeline on Snowflake?

Define each stage as an asset in one project and let the framework resolve the order. With Bruin, ingestion, SQL and Python transformations, and quality checks are all assets, and bruin run executes the graph against Snowflake. Loads use an internal stage plus COPY INTO.

Do I need an orchestrator to run Snowflake pipelines?

Not for a straightforward pipeline. bruin run resolves dependencies itself, so a CI runner on a schedule is enough. A dedicated orchestrator earns its keep once you need complex retries, backfills, and cross-team scheduling.

What is the cheapest way to run pipelines on Snowflake?

On Snowflake the main lever is warehouse auto-suspend and right-sizing, since idle compute is the usual overspend. Incremental models using MERGE on a primary key matter more than which tool you pick.

About Bruin

What is Bruin?

Bruin is an open-source, AI-powered data platform. One project covers the whole path from source to answer: data ingestion (with ingestr), SQL and Python transformations, quality checks, column-level lineage, a data catalog and scheduling. On top of that data, an AI data analyst answers questions in Slack, Microsoft Teams, Google Chat, WhatsApp, Discord, Telegram, email or the browser, and builds dashboards from a prompt.

Pipelines are plain SQL, Python and YAML files in your own Git repository, and the work runs inside your own warehouse: Snowflake, BigQuery, Databricks, Redshift, Postgres, ClickHouse, DuckDB and others. The Bruin CLI is open source under Apache 2.0 and runs on a laptop, in CI or on-prem. Bruin Cloud is optional and adds scheduling, the AI data analyst, governance, SSO and audit logs.

For warehouse pipelines that means SQL and Python transformations in one project, run inside your own warehouse, with checks and tests next to the code.

Replaces
Separate tools for ingestion, transformation, orchestration, data quality, cataloguing and BI
Open source
Bruin CLI under Apache 2.0; ingestr source-available under the Functional Source License
Your data
Stays in your warehouse; pipelines stay in your Git repository
Security
ISO/IEC 27001:2022 certified and SOC 2 Type 2 attested
Used by
Data teams at Lessmore, StarBerry, NativeMinds, SCG, M8 Games and Fabrikatör

Why Bruin for warehouse pipelines

  • SQL and Python together

    SQL and Python assets run in the same pipeline, share lineage and are tested with the same checks.

  • Runs in your warehouse

    Transformations run on Snowflake, BigQuery, Databricks, Redshift, Postgres, ClickHouse, DuckDB and others. Data does not leave your infrastructure.

  • Incremental by design

    Table, view, merge, delete+insert and append materializations keep runs small as tables grow.

  • Tested before it ships

    bruin validate checks the whole pipeline, and unit tests run each query against mocked inputs.

How it works

  1. 01

    Start from a template

    Run bruin init to create a project for your warehouse.

  2. 02

    Write assets

    Write SQL or Python assets with their columns, checks and materialization in the same file.

  3. 03

    Validate and run

    Run bruin validate, then bruin run, and schedule the pipeline when it works.

One pipeline, end to end

Bruin CLI and ingestr are on GitHub.

A demo walks through your own data.

Sign up to our newsletter

Practical updates on open-source data pipelines, AI analysts, governance, and what we are shipping at Bruin.

The signup form is hosted by Brevo. Allow marketing cookies to load it.