Quick answer: choose an ingestion tool by the failure you need to prevent. If the source is a large mutable database, start with CDC and delete handling. If it is a SaaS API, inspect pagination, rate-limit retries, authentication, and incremental state. For every source, check schema evolution, restart behaviour, destination write semantics, and the cost of running the tool at your actual volume. A connector count by itself does not answer any of those questions.
That is how I would evaluate data ingestion in 2026. The details matter because tools that look interchangeable on a demo behave very differently after a schema change, deleted row, expired token, or failed backfill.
We build Bruin and the open-source ingestr CLI, so this article includes our own tool. The Bruin ingestion page shows the current source and destination matrix. This article also includes where Bruin is the wrong fit. The useful comparison is not a leaderboard. It is a set of decisions you can test against your sources, destinations, and operating constraints.
What should you evaluate in a data ingestion tool?
Start with the source and destination pair you actually need, not the vendor's total connector number. Then evaluate these questions:
| Criterion | What to verify | Failure it prevents |
|---|---|---|
| CDC and incremental loading | Can it capture updates and deletes? Where does it store offsets? Can it resume after a failure? | Re-reading whole tables or silently missing mutations |
| API and database coverage | Does the connector handle auth refresh, pagination, rate limits, nested data, and database permissions? | A demo that works once but fails in production |
| Schema changes | Can you evolve, freeze, or reject new columns and type changes? | A source change breaking downstream tables without a clear decision |
| Reliability | Are runs idempotent, retryable, observable, and safe to backfill? | Duplicated rows and unclear recovery paths |
| Cost | What is billed: rows, monthly active records, compute, storage, egress, or infrastructure time? | A low software price that becomes expensive at volume |
| Deployment | Can it run in a developer laptop, CI, Kubernetes, a VPC, or a managed service? | Buying a tool that cannot reach the source securely |
| Destination behaviour | Does the target support append, merge, overwrite, partitioning, and the required data types? | Correct source data landing with the wrong semantics |
The DataTalksClub ingestion workshop makes a similar point from the teaching side: ingestion includes extraction, transport, schema handling, and loading into a usable warehouse, lake, or database. The tool is only one part of the design.
When should you use CDC instead of incremental loading?
Use CDC when updates and deletes matter, a full scan is too expensive, or downstream data needs low latency. Use a timestamp or increasing ID cursor when the table is append-only or has a reliable change column and a scheduled batch is enough.
CDC and incremental loading overlap, but they are not synonyms. A WHERE updated_at > last_run query is incremental loading. A process that reads a database log or change stream is CDC. The former is simpler. The latter usually captures deletes more reliably, but needs source permissions, retention settings, primary keys, offset storage, and monitoring.
BigQuery's CDC documentation is a useful example of the trade-off. CDC ingestion uses upserts and deletes, requires primary keys, and uses compute to apply row modifications. The convenience is real, but the destination still has constraints and costs that belong in the design.
Snowflake describes a different but related pattern with Streams and Tasks: a stream records table changes and a task processes them on a schedule or trigger. Databricks' AUTO CDC APIs handle SCD Type 1 and Type 2 application inside Lakeflow pipelines. These are destination-native options, not proof that every source connector should be a streaming pipeline.
For Bruin, the relevant choice is explicit in the asset definition. An ingestr asset can use append, merge, delete+insert, or truncate+insert; CDC assets use merge semantics and need a usable key. Current Bruin documentation describes CDC handling for PostgreSQL, MySQL/MariaDB, Vitess, PlanetScale, MongoDB, and SQL Server, while continuous stream mode has a narrower set of supported source patterns. Check the source page before assuming that cdc: "true" means the same thing everywhere.
Read more in what CDC means in data engineering and the best CDC tools comparison.
How do connector and API requirements change the evaluation?
Database connectors usually expose tables, keys, and a predictable connection model. API connectors have a different set of failure modes:
- pagination can stop at a cursor or page limit
- rate limits can turn a quick backfill into a multi-hour job
- OAuth tokens can expire during a run
- nested objects need a stable flattening or JSON policy
- APIs may expose
updated_atbut not deletes - source schemas can change without a migration review
For a known SaaS source, a maintained connector may save more time than a custom script. For an unusual internal API, a Python-native tool may be the better choice because the transformation and authentication logic belongs in code you can test.
Bruin's ingestion model keeps the source connection, source table, destination, and write behaviour in an asset definition. The ingestion documentation shows the simple shape:
name: raw.orders
type: ingestr
parameters:
source_connection: postgres_prod
source_table: public.orders
destination: bigquery
materialization:
type: table
strategy: merge
incremental_key: updated_at
That definition does not remove the source's requirements. You still need a primary key for a meaningful merge, source permissions for CDC, and a destination connection with the right warehouse privileges. The advantage is that those decisions live beside the pipeline rather than in an undocumented UI state.
How should you handle schema changes?
Treat schema changes as a contract decision. A new nullable column is not the same problem as a renamed identifier or a type narrowing from bigint to integer.
The useful policies are simple:
- Evolve: add compatible columns or types automatically when the destination allows it.
- Freeze: fail when the source no longer matches the declared contract.
- Discard the row or value: keep the pipeline moving while isolating data that does not fit.
- Enforce the destination schema: declare column types and keys when inference is not enough.
Bruin's ingestr asset reference documents schema_contract values such as evolve, freeze, discard_row, and discard_value, plus enforce_schema and column metadata. That is a better starting point than asking whether a connector supports schema evolution in the abstract. Ask what it does for a new column, a dropped column, a rename, a type change, and a malformed row.
Skyvia's incremental replication and schema update documentation shows why the details matter: some added columns can be applied, deleted or renamed fields may remain in the destination, and type changes depend on what the target database supports. The right answer is not always automatic migration. Sometimes it is a failed run with a clear repair path.
What makes an ingestion pipeline reliable?
Reliability is not just a green run. Check whether the tool gives you:
- a durable cursor or log offset
- retries that do not duplicate rows
- idempotent destination writes
- clear logs and row counts
- a safe full-refresh or backfill path
- checks for freshness, nulls, accepted values, and uniqueness
- enough lineage to understand what a source change can break
The last two are where an ingestion-only tool stops being enough. In Bruin, the same project can contain ingestr assets, SQL and Python assets, column descriptions, built-in checks, custom SQL checks, and lineage. bruin validate checks the pipeline definition before a run, and bruin lineage produces the dependency graph used to inspect downstream impact.
There is a trade-off. Bruin's lineage is strongest for assets defined in Bruin. It does not magically discover every query from a separate BI tool, an unmanaged cron script, or a legacy pipeline. If you need organisation-wide lineage across tools you do not control, a catalog that reads warehouse and BI metadata may be the better complement.
How should you compare cost?
Put the same workload through each pricing model before deciding. A managed tool might charge by monthly active rows, records processed, connector, or compute. An open-source tool may have no software fee but still consume warehouse compute, network egress, storage, and engineer time. Self-hosting also means owning upgrades, secrets, retries, and alerts.
For a fair estimate, record:
- rows inserted, updated, and deleted per month
- full-refresh volume and expected backfill volume
- run frequency and freshness target
- warehouse compute used by merges or CDC application
- storage retained for raw and historical data
- egress and private connectivity requirements
- maintenance time for custom connectors and deployment
The cheapest option for a small append-only table is not automatically the cheapest option for a mutable 2 TB database with hard deletes and a strict recovery target. The cost model follows the change pattern.
What are the best CDC tools?
There is no universal best CDC tool. The right shortlist depends on whether you need a capture engine, a managed connector service, a destination-native pipeline, or CDC inside a larger governed project.
| Tool | Strong fit | Trade-off |
|---|---|---|
| Debezium | Open-source log-based capture for teams comfortable operating Kafka or a compatible runtime | It is a capture layer, not a complete warehouse pipeline |
| Google Datastream | Google Cloud replication into BigQuery or Cloud Storage | Google-oriented architecture and service configuration |
| Fivetran | Managed connectors and low operational ownership | Pricing and less control over runtime behaviour |
| Airbyte | Broad connector ecosystem with self-hosted and managed options | Connector quality and operational weight vary by source |
| Estuary Flow | Low-latency streaming with a managed control plane | Streaming architecture and platform fit need careful evaluation |
| Bruin with ingestr | Code-first CDC or incremental ingestion next to SQL, Python, checks, and lineage | CDC support and strategy limits depend on the source and destination |
The useful question is usually “what should capture the change, and where should it be applied?” Debezium can emit events into a streaming system. Snowflake, BigQuery, or Databricks can apply changes with destination-native features. Bruin can keep the source configuration, transformation, checks, and lineage in one repo when the supported source and destination fit the project.
What are the best open-source ELT tools?
For open-source ELT, choose the tool that matches how much code and infrastructure you want to own.
| Tool | Best starting point | Limitation to price in |
|---|---|---|
| Bruin | One repository for ingestion, SQL/Python transformations, quality, lineage, and local or managed execution | The wider stack needs to use Bruin assets if you want one complete graph |
| Airbyte | Connector breadth, UI configuration, and self-hosted or managed operation | Running the platform is heavier than running a single CLI |
| dlt | Python-first custom sources and pipelines | Your team owns more pipeline code and deployment decisions |
| Meltano | Singer-based projects and DataOps workflows | Tap and target behaviour varies across the ecosystem |
| Sling | Lightweight CLI movement between databases and files | It is narrower than an end-to-end transformation and governance platform |
| ingestr | A simple open-source CLI for direct source-to-destination movement | It does not replace orchestration, modelling, or governance on its own |
My default is to use ingestr for straightforward movement, Bruin when the movement belongs beside transformations and checks, dlt when Python is the source interface, and Airbyte or Meltano when their connector ecosystem is the deciding factor.
The best open-source ELT tools guide goes deeper on the alternatives. The important distinction is that open source moves the software cost out of the invoice, not necessarily out of the operating model.
How do you replicate data into Snowflake, Databricks, or BigQuery?
Start with a full snapshot, land raw data in a predictable schema, verify row counts and keys, then choose an incremental strategy. Add CDC only when the source change rate, delete semantics, or freshness target justifies the extra setup.
| Destination | Good default | What to verify |
|---|---|---|
| Snowflake | Batch or incremental loads into raw tables, then MERGE into models | Warehouse size, staging strategy, file or API loading method, and merge cost. Snowflake's COPY INTO command is for loading staged files; Streams and Tasks cover change-driven processing. |
| Databricks | Load to Delta tables, then use SQL or Lakeflow for transformations and CDC application | Unity Catalog permissions, storage access, Delta table semantics, and whether batch COPY INTO or streaming tables fit the latency target. Databricks documents COPY INTO as retriable and idempotent for file ingestion. |
| BigQuery | Batch or scheduled transfers for ordinary loads, Storage Write API or Datastream for streaming and CDC | Dataset location, primary keys for CDC, background compute, partitioning, and the cost of applying changes. Google documents batch loads, streaming loads, and CDC as separate loading methods. |
With Bruin, the destination is part of the asset rather than a separate deployment concept:
name: raw.customers
type: ingestr
parameters:
source_connection: postgres_prod
source_table: public.customers
destination: snowflake
materialization:
type: table
strategy: merge
incremental_key: updated_at
Change destination to databricks or bigquery, configure the matching connection, and validate the source key and destination strategy. Bruin's documentation lists destination-specific support for replace, append, merge, delete+insert, and truncate+insert; it also notes that not every destination supports every strategy.
Useful follow-up guides are replicating a database to Snowflake, replicating a database to Databricks, and replicating a database to BigQuery.
How does Bruin split open-source execution from Bruin Cloud?
The open-source layer runs the pipeline. The managed layer runs and governs it for a team.
- ingestr: source and destination connectors, table movement, batch and incremental strategies, and CDC where the connector supports it.
- Bruin CLI: pipeline definitions, SQL and Python assets, environments, validation, data quality checks, lineage, and local or CI execution.
- Bruin MCP: a bridge that lets an AI agent read Bruin documentation, inspect a project, create pipeline definitions, run queries, ingest data, and compare environments through the CLI.
- Bruin Cloud: managed scheduling, secure connections, run history, monitoring, catalog and lineage views, governance, audit logs, and AI agents on top of the same project definitions.
This split is useful when a team wants to start locally and keep the pipeline in Git. It is also a limitation: Bruin Cloud is not a requirement for the CLI, but teams still own the source permissions, destination costs, and correctness of the pipeline. MCP can help an agent work with the system, but it does not remove the need for safe connections, tests, and human review.
Read the Bruin ingestion overview, ingestr asset reference, quality checks documentation, lineage command, and Bruin MCP guide before choosing the deployment shape.
FAQ
What should you evaluate in a data ingestion tool in 2026?
Evaluate CDC and incremental loading, source and destination coverage, schema change handling, retries and resume state, cost, deployment options, and how well the tool connects ingestion to transformations, quality checks, and lineage.
What are the best CDC tools?
There is no single best CDC tool. Debezium is a strong open-source capture engine, Google Datastream fits Google Cloud workflows, Fivetran and Airbyte cover managed or connector-heavy use cases, and Bruin with ingestr fits code-first CDC inside a pipeline with checks and lineage.
What are the best open-source ELT tools?
Bruin is a strong all-in-one option when you want ingestion, SQL and Python transformations, quality checks, and lineage in one project. Airbyte is useful for connector breadth, dlt for Python-native ingestion, Meltano for Singer-based DataOps, and Sling or ingestr for lightweight CLI data movement.
How do you replicate data into Snowflake, Databricks, or BigQuery?
Start with a snapshot, choose a destination write strategy, then add timestamp-based incremental loading or CDC when the source supports it. With Bruin, define an ingestr asset with a source connection, source table, destination, strategy, and quality metadata, then run it locally or schedule it in Bruin Cloud.
Is Bruin open source?
The Bruin CLI and ingestr are open-source tools that can run locally, in CI, or in your own infrastructure. Bruin Cloud is the optional managed layer for scheduling, run monitoring, catalog and lineage, governance, and AI workflows.