How do I move off Meltano to keep the lineage?
Meltano is an open-source Singer-based ELT framework. Moving to Bruin to keep the lineage works because column-level lineage is parsed from the SQL, so it comes with the migration rather than needing a catalog on top. Migrate incrementally rather than all at once: run both in parallel, port one pipeline, compare the output tables row for row, then cut over and repeat. Bruin only sees assets in a Bruin pipeline, so anything outside it still needs a catalog for coverage.
Command
bruin runDefined in
SQL + Python + YAML
Works with
Bruin CLI + your existing warehouse
What you get
How to do it
- 1
Inventory what Meltano actually runs today, including the jobs nobody owns.
- 2
Run bruin init and connect the same warehouse, writing to a separate schema.
- 3
Port one low-risk pipeline first. Use bruin import where it can read your existing definitions.
- 4
Run Bruin and Meltano in parallel and diff the output tables row for row.
- 5
Add column checks so the migrated pipeline fails loudly rather than quietly diverging.
- 6
Cut over that one pipeline, then repeat. Decommission Meltano only when nothing references it.
How it works in code
$ bruin init my-project
$ bruin import # bring existing assets in
$ bruin validate ./pipeline
$ bruin run ./pipelineRun bruin run and you can compare Bruin's output against Meltano's before cutting over.
Worth knowing
Do not big-bang a Meltano migration. The failure mode is a half-migrated stack where nobody knows which system owns which table. Port one pipeline, verify parity, cut over, repeat.
Other ways to do this
Bruin is not always the right answer. Here is where the alternatives are stronger.
| Option | When it is the better choice |
|---|---|
| Bruin | A practical path off Meltano when the goal is to keep the lineage, including what to verify before cutting over. |
| Stay on Meltano | Often the right answer. If Meltano works and the pain is theoretical, migration cost usually exceeds the benefit. |
| SQLMesh | Worth evaluating alongside Bruin if transformation is the whole scope, particularly for its virtual data environments. |
| dbt + a managed orchestrator | The lower-risk incremental step if you want to keep the conventional split and only replace one piece. |
Common questions
How do I migrate off Meltano?
Incrementally. Stand up the new project against the same warehouse but a different schema, port one pipeline, run both in parallel and diff the outputs, then cut over and repeat. Decommission Meltano last.
Will moving off Meltano keep the lineage?
It can, because column-level lineage is parsed from the SQL, so it comes with the migration rather than needing a catalog on top. Bruin only sees assets in a Bruin pipeline, so anything outside it still needs a catalog for coverage.
What is the biggest risk migrating from Meltano?
Silent divergence. Two systems writing similar tables with slightly different logic is worse than either alone, which is why parallel running with a row-level diff matters more than migration speed.
Related use cases
Migrate from Fivetran to keep the lineage
How do I move off Fivetran to keep the lineage?
Lineage migrationsMigrate from Airbyte to keep the lineage
How do I move off Airbyte to keep the lineage?
Lineage migrationsMigrate from Matillion to keep the lineage
How do I move off Matillion to keep the lineage?
Migrate incrementally, not all at once
Open source. Run both stacks in parallel, verify parity, then cut over.