Lineage/Lineage by warehouseData Platform Engineer

A Databricks metric looks wrong. Where did it go bad?

Walk the Databricks column lineage upstream from the metric to inspect each transformation and narrow down where the value diverged. Bruin does this in one platform: ingestion, SQL and Python pipelines, quality checks, lineage, and an AI data analyst that answers in Slack, Microsoft Teams, Google Chat, WhatsApp, Discord, Telegram, email and the browser.

Command

bruin lineage

Defined in

SQL + YAML

Works with

Databricks + Bruin CLI

What you get

Debuggingemergentcolumn-level

How it works in code

$ bruin lineage ./pipeline --downstream  # databricks

Run bruin lineage and you see the full Databricks column graph before changing anything.

About Bruin

What is Bruin?

Bruin is an open-source, AI-powered data platform. One project covers the whole path from source to answer: data ingestion (with ingestr), SQL and Python transformations, quality checks, column-level lineage, a data catalog and scheduling. On top of that data, an AI data analyst answers questions in Slack, Microsoft Teams, Google Chat, WhatsApp, Discord, Telegram, email or the browser, and builds dashboards from a prompt.

Pipelines are plain SQL, Python and YAML files in your own Git repository, and the work runs inside your own warehouse: Snowflake, BigQuery, Databricks, Redshift, Postgres, ClickHouse, DuckDB and others. The Bruin CLI is open source under Apache 2.0 and runs on a laptop, in CI or on-prem. Bruin Cloud is optional and adds scheduling, the AI data analyst, governance, SSO and audit logs.

For lineage and governance that means column-level lineage, owners and a glossary that come from the pipeline code itself, so they stay current without a separate catalog to feed.

Replaces
Separate tools for ingestion, transformation, orchestration, data quality, cataloguing and BI
Open source
Bruin CLI under Apache 2.0; ingestr source-available under the Functional Source License
Your data
Stays in your warehouse; pipelines stay in your Git repository
Security
ISO/IEC 27001:2022 certified and SOC 2 Type 2 attested
Used by
Data teams at Lessmore, StarBerry, NativeMinds, SCG, M8 Games and Fabrikatör

Why Bruin for lineage and governance

  • Column-level lineage from the SQL

    Bruin extracts column lineage from SQL assets automatically. Python assets declare their upstream columns.

  • Across pipelines

    Asset URIs connect pipelines and repositories, and Bruin Cloud shows every pipeline and the dependencies between them.

  • Owners and meaning on every column

    Owners, descriptions, domains, tags and glossary terms live in the same files as the code.

  • Impact before you change

    See what is downstream of a table or column before you rename or drop it.

How it works

  1. 01

    Write the pipeline

    Write SQL and Python assets as usual; dependencies come from the SQL.

  2. 02

    Inspect it locally

    Run bruin lineage --full on any asset to see everything upstream and downstream.

  3. 03

    See it across teams

    Connect the repository to Bruin Cloud for the lineage view across every pipeline.

Know what breaks before you ship it

Bruin CLI and ingestr are on GitHub.

A demo walks through your own data.

Sign up to our newsletter

Practical updates on open-source data pipelines, AI analysts, governance, and what we are shipping at Bruin.

The signup form is hosted by Brevo. Allow marketing cookies to load it.