Now available · 12x faster than the rest

Meet ingestr v1.
Ingestion, rebuilt for speed.

View on GitHub Read the docs

Open source
MIT license
★3.5K stars

ingestr v1 · By Bruin0.00s

dlt 0.00s

sling 0.00s

Works with

// why.ingestr-v1

Built to be the fastest path from source to destination.

ingestr v1 keeps everything teams already love about ingestr: a single binary, declarative sources and destinations, incremental and full-refresh modes, and a CLI you can drop into any pipeline. What's new is what happens between the source and the destination.

Up to 12x faster end-to-end: Across five common source-destination pairs (duckdb, postgres, mysql in any direction), ingestr v1 is the fastest tool we measured in every single one.
Drop-in upgrade from v0: Same command surface, same connectors, same place in your pipelines. If your jobs run on ingestr today, they'll run on v1 tomorrow.
Open source first: ingestr is open source alongside the Bruin CLI. Bruin Cloud adds orchestration, governance, and observability on top.

// benchmarks.full-suite

Fastest in every scenario we ran.

Each row is the mean of 5 runs loading a 1M-row table from source to destination on identical hardware. v0 is included for reference.

Scenario	ingestr v1	sling	dlt	ingestr v0
duckdb → duckdb	4.34s	48.15s	18.56s	13.66s
duckdb → postgres	4.18s	48.04s	33.93s	31.43s
mysql → postgres	4.98s	28.18s	53.08s	52.38s
postgres → duckdb	6.08s	39.11s	26.79s	21.42s
postgres → postgres	6.14s	20.97s	44.44s	41.11s

// strategies

Six ways to load your data.

Pick the incremental strategy that fits the table you're moving. Two are new in v1.

replaceDefault. Swap the destination table for the source.: Download the source and overwrite the destination. v1 uses double buffering under the hood to keep transaction times short on large loads.; No keys required
appendAdd new rows only, using an incremental key.: Filter the source by an incremental key (e.g. `updated_at`) and append rows newer than the last run. Ideal for event streams and audit logs where history matters.; Requires `incremental_key`
mergeUpsert by primary key. Update existing rows, insert new ones.: Match rows by `primary_key`, update changed values, and insert anything new. The standard CDC pattern for evolving entities.; Requires `primary_key` + `incremental_key`
delete+insertDelete the matching window, then insert fresh.: Delete destination rows that fall in the incremental window and re-insert from source. The right call for late-arriving data or backfills over a fixed range.; Requires `incremental_key`
truncate+insertNew in v1. Empty the table and load fresh in one go.: Truncate the destination table and load the new rows in a single operation. Faster than full replace when you do not need the staging dance, and safer than a manual TRUNCATE + INSERT.; No keys required
scd2New in v1. Slowly Changing Dimension Type 2 with row history.: Keep a full history per primary key with `valid_from` / `valid_to` columns, so you can query the state of any record at any point in time. The right pattern for evolving dimensions like customers, products, or accounts.; Requires `primary_key`

// get.started

One binary. One command.

Install ingestr v1 and move your first table in under a minute.

terminal

$ brew install bruin-data/tap/ingestr

$ ingestr run \
    --source postgres://user:***@host/db \
    --dest   duckdb:///warehouse.db \
    --table  public.orders

→ loaded 1,000,000 rows in 4.18s

View on GitHub Read the docs

FREQUENTLY ASKED

What's new in ingestr v1?

What is actually new in ingestr v1?+

Same CLI, same flags, same source and destination URIs. Under the hood, v1 is a ground-up rewrite with substantially lower memory use and meaningfully better throughput on most workloads. It adds two new incremental strategies (truncate+insert and scd2), reworks replace with double buffering, and adds new sources and destinations.

Is ingestr v1 backwards compatible with v0?+

For the common path, yes. The CLI surface, source and destination URIs, and the existing strategies behave the same way. The migration guide covers the handful of intentional changes (the biggest one: nested JSON is kept as a JSON column instead of being flattened into separate columns).

Do I need to migrate my existing ingestr pipelines?+

Reinstall ingestr and most pipelines run unchanged. Two things to check before you flip over: any downstream queries that relied on flattened JSON columns (those are now a single JSON column you select into), and BigQuery sources, which now need the roles/bigquery.readSessionUser IAM role because v1 reads via the Storage Read API.

Which incremental loading strategies are supported?+

Six total: replace (default), append, merge, delete+insert, truncate+insert (new in v1), and scd2 (new in v1). Each strategy is documented with the keys it requires and when to use it on the ingestr docs.

Does ingestr support Slowly Changing Dimensions?+

Yes, SCD Type 2 is built in as the scd2 strategy in v1. Provide a primary_key and ingestr maintains valid_from / valid_to columns so you can query the historical state of any record. No extra modeling, no custom merge SQL.

Which sources and destinations are supported?+

BigQuery, Snowflake, Databricks, Redshift, Postgres, MySQL, MS SQL, ClickHouse, DuckDB, MotherDuck, MongoDB, DynamoDB, Elasticsearch, SQLite, Trino, and many more on the database side, plus a long list of SaaS sources (Stripe, HubSpot, Salesforce, Google Ads, Facebook Ads, GitHub, Notion, Airtable, Klaviyo, and dozens more). The full matrix lives in the ingestr README.

How many connectors ship out of the box?+

22 databases and warehouses (each available as both source and destination, with the few exceptions noted in the README) plus 40+ SaaS platforms as sources. For anything not built in, the Bruin platform can pipe through MCP, giving you instant reach into thousands of additional services without writing a custom connector.

Is there a web UI, or is it CLI only?+

Both. ingestr server starts a local web UI that lets you save reusable connections, configure a run, stream logs while it executes, and review previous runs. Backed by SQLite for run history. The CLI is still the primary surface for production pipelines.

Can I run ingestr in Docker, CI/CD, or Kubernetes?+

Yes. ingestr ships as a single binary with a single-command install, so it drops cleanly into GitHub Actions, GitLab CI, Argo, Airflow, Dagster, Prefect, k8s jobs, or a plain cron. Most production users invoke it from CI or an orchestrator rather than running it interactively.

How was "up to 12x faster" measured?+

Mean over 5 runs loading 1M-row tables across five common source-destination pairs (duckdb, postgres, mysql in either direction), on identical hardware, compared against ingestr v0, dlt, and sling. The 12x figure compares ingestr v1 to the slowest tool in each scenario.

Is ingestr open source?+

Yes. ingestr is MIT-licensed (some components Apache 2.0) and lives at github.com/bruin-data/ingestr alongside the rest of Bruin's open-source CLI. You can self-host it anywhere a binary runs. Bruin Cloud is the optional managed layer on top, not a requirement.

What does ingestr collect about my usage?+

Anonymous telemetry only: a hashed machine ID, ingestr version, OS / architecture, the command that ran, and whether it succeeded. No connection strings, no schemas, no data. Disable it completely by setting INGESTR_DISABLE_TELEMETRY=true.

How does ingestr fit with Bruin Cloud?+

Bruin Cloud orchestrates ingestr jobs alongside SQL/Python transformations and quality checks, with column-level lineage, audit logs, cost insights, and an AI data analyst on top. ingestr stays a standalone open-source CLI; the Cloud is opt-in.

Where do I get help or report a bug?+