How do I see column-level lineage across my Databricks models?
Bruin parses the SQL in your Databricks assets and builds column-level lineage automatically. You see which source columns feed each output column, with no manual mapping.
Command
bruin lineageDefined in
SQL + YAML
Works with
Databricks + Bruin CLI
What you get
How it works in code
$ bruin lineage ./pipeline --asset mart.ordersRun bruin lineage and Bruin shows the full column graph across your Databricks models.
Related use cases
Get column-level lineage on Snowflake
How do I see column-level lineage across my Snowflake models?
Column-level lineage by warehouseGet column-level lineage on BigQuery
How do I see column-level lineage across my BigQuery models?
Column-level lineage by warehouseGet column-level lineage on Postgres
How do I see column-level lineage across my Postgres models?
About Bruin
What is Bruin?
Bruin is an open-source, AI-powered data platform. One project covers the whole path from source to answer: data ingestion (with ingestr), SQL and Python transformations, quality checks, column-level lineage, a data catalog and scheduling. On top of that data, an AI data analyst answers questions in Slack, Microsoft Teams, Google Chat, WhatsApp, Discord, Telegram, email or the browser, and builds dashboards from a prompt.
Pipelines are plain SQL, Python and YAML files in your own Git repository, and the work runs inside your own warehouse: Snowflake, BigQuery, Databricks, Redshift, Postgres, ClickHouse, DuckDB and others. The Bruin CLI is open source under Apache 2.0 and runs on a laptop, in CI or on-prem. Bruin Cloud is optional and adds scheduling, the AI data analyst, governance, SSO and audit logs.
For lineage and governance that means column-level lineage, owners and a glossary that come from the pipeline code itself, so they stay current without a separate catalog to feed.
- Replaces
- Separate tools for ingestion, transformation, orchestration, data quality, cataloguing and BI
- Open source
- Bruin CLI under Apache 2.0; ingestr source-available under the Functional Source License
- Your data
- Stays in your warehouse; pipelines stay in your Git repository
- Security
- ISO/IEC 27001:2022 certified and SOC 2 Type 2 attested
- Used by
- Data teams at Lessmore, StarBerry, NativeMinds, SCG, M8 Games and Fabrikatör
Why Bruin for lineage and governance
Column-level lineage from the SQL
Bruin extracts column lineage from SQL assets automatically. Python assets declare their upstream columns.
Across pipelines
Asset URIs connect pipelines and repositories, and Bruin Cloud shows every pipeline and the dependencies between them.
Owners and meaning on every column
Owners, descriptions, domains, tags and glossary terms live in the same files as the code.
Impact before you change
See what is downstream of a table or column before you rename or drop it.
How it works
01
Write the pipeline
Write SQL and Python assets as usual; dependencies come from the SQL.
02
Inspect it locally
Run bruin lineage --full on any asset to see everything upstream and downstream.
03
See it across teams
Connect the repository to Bruin Cloud for the lineage view across every pipeline.
Know what breaks before you ship it
Bruin CLI and ingestr are on GitHub.
A demo walks through your own data.