What is CDC in data engineering? Learn how change data capture records database changes, differs from ETL and incremental loading, and when to use it.
Arsalan Noorafkan
Developer Advocate
Quick answer: CDC stands for change data capture. It is a way to track inserts, updates, and deletes in a source system, then move those changes into another system without reloading the whole table every time.
The shortest version:
source database changes -> CDC process -> warehouse, lake, stream, or another database
If an order is inserted in Postgres, CDC can send that new order to BigQuery. If a customer row is updated in MySQL, CDC can send the updated version to Snowflake. If an account is deleted in SQL Server, CDC can preserve that delete as an event so the destination does not quietly drift away from the source.
That's the useful bit. CDC is not just "faster ETL". It changes the shape of the ingestion problem: instead of asking "what does the full table look like now?", you ask "what changed since the last point I safely processed?"
This article covers the broader capture method. For the continuous, low-latency delivery model, see CDC streaming.
For data teams, this matters because most company data does not sit still. Product events arrive, payments settle, subscriptions change status, tickets get merged, rows get corrected, and someone eventually deletes the thing everyone assumed would never be deleted. The classic nightly full refresh works for small tables. It gets painful once tables are large, freshness matters, or downstream systems need a proper history of changes.
Change data capture is the process of detecting row-level changes in a source system and making those changes available to another system.
A CDC event usually answers a few practical questions:
A simplified CDC event might look like this:
{
"table": "public.orders",
"operation": "update",
"primary_key": { "order_id": 1042 },
"before": {
"status": "pending",
"updated_at": "2026-07-13T08:14:10Z"
},
"after": {
"status": "paid",
"updated_at": "2026-07-13T08:16:03Z"
},
"source_position": "log-sequence-number-or-binlog-offset"
}
That source_position part is easy to ignore when you are learning CDC, but it is one of the most important pieces. A CDC pipeline needs to know where it stopped so it can resume without skipping changes or processing the same change incorrectly. In PostgreSQL, this is tied to logical decoding and replication slots. In MySQL, it is commonly tied to the binary log. In SQL Server, CDC reads from the transaction log into change tables.
The names vary by database, but the pattern is similar: the database already records changes so it can commit transactions, recover from failure, and replicate data. CDC uses that record as the source of truth for downstream movement.
There are a few ways to capture changes, but log-based CDC is the one most engineers mean when they say CDC.
An application inserts, updates, or deletes a row. The database commits the transaction and records it in its internal log.
Examples:
The CDC reader connects to the source database and reads committed changes in order. It does not need to run select * from orders every five minutes. It reads the database's change stream.
This usually reduces source load compared with full-table extraction, especially when a table is large but only a small percentage of rows change during each sync.
The pipeline records the last safe position it processed. That offset might be a log sequence number, binlog file and position, timestamp plus version, or another source-specific cursor.
This is what lets the pipeline restart after a deploy, network failure, worker crash, or source maintenance window.
The destination might receive the changes as:
Analytics teams often keep both:
That split is boring, but it saves you later. Raw events are useful when something goes wrong and you need to replay or explain history. Current tables are useful because most business users want to query "orders" rather than reconstruct a table from every historical mutation.
CDC is not a replacement for ETL or ELT. It is a method of extraction.
| Concept | What it means | Where CDC fits |
|---|---|---|
| ETL | Extract, transform, load | CDC can be the extract step |
| ELT | Extract, load, transform | CDC can load raw changes first, then transform in the warehouse |
| Data ingestion | Moving data from source to destination | CDC is one ingestion strategy |
| Replication | Keeping another system close to the source | CDC may power replication, but it is not limited to source-shaped copies |
| Streaming | Moving events continuously | CDC can produce a stream, but CDC can also be consumed in batches |
The difference sounds pedantic until you design the pipeline.
If you say "we need ETL", you still have to decide whether extraction is full refresh, timestamp incremental, API pagination, event export, or CDC. If you say "we need CDC", you still have to decide what happens after the changes land: raw storage, merges, transformations, checks, lineage, documentation, and access control.
This is why CDC is usually one part of a broader data platform, not the whole platform.
CDC and incremental loading overlap, but they are not the same thing.
Incremental loading means "only load new or changed data since the last run." CDC is one way to do that.
The common incremental strategies look like this:
| Strategy | How it finds changes | Good for | Main risk |
|---|---|---|---|
| Timestamp cursor | Reads rows where updated_at > last_run_time | Simple tables with reliable timestamps | Missed deletes, clock issues, late updates |
| Increasing ID | Reads rows where id > last_seen_id | Append-only tables | Cannot detect updates or deletes |
| Full diff | Compares source and destination snapshots | Smaller tables where correctness matters | Expensive at scale |
| Log-based CDC | Reads transaction logs or change streams | Large mutable tables, low-latency syncs | More setup, permissions, and operational detail |
| Trigger-based CDC | Writes changes through database triggers | Databases without usable logs | Adds write-path overhead and schema complexity |
If a table is append-only, an increasing ID cursor might be enough. If rows are updated and deleted all day, CDC becomes much more attractive.
The mistake is treating every table the same. A small reference table can be full-refreshed. A huge orders table probably should not be. An event table may only need append mode. A customer table with merges, deletes, and compliance requirements may need CDC or SCD Type 2.
This is the usual preferred option for production databases. The CDC tool reads the transaction log, binlog, write-ahead log, redo log, or logical replication stream.
Why teams like it:
The tradeoff: you need source-specific setup. Permissions, retention, replication slots, binlog settings, replica identity, networking, and failover behaviour matter.
Trigger-based CDC adds database triggers that write changes to audit or shadow tables. A pipeline then reads those tables.
This can work, especially when the database does not expose a useful log. But it adds logic to the write path. That means more performance risk, more migration work, and more things to review when schema changes happen.
Query-based CDC polls the source with a cursor, usually updated_at, created_at, or an incrementing ID.
This is the easiest model to understand and often the first one teams implement. It is also the easiest one to get subtly wrong.
The usual problems:
Query-based incremental loading is still useful. It just should not be called "proper CDC" unless it really captures the mutations you care about.
CDC is worth considering when at least one of these is true:
Good CDC use cases:
CDC is especially useful when the business keeps asking why yesterday's dashboard is not fresh enough, but the data team knows that full-refreshing the source every 15 minutes would be irresponsible.
If you search for "CDC tools", you will usually see the same names come up: Debezium, Fivetran, Airbyte, Estuary Flow, AWS Database Migration Service, Google Datastream, Oracle GoldenGate, Qlik Replicate, and now Bruin with ingestr for code-first CDC inside a governed data pipeline.
They are not interchangeable. Some are CDC engines. Some are managed ELT platforms. Some are cloud migration services that happen to support ongoing replication. Some are better when Kafka is already part of your stack. Some are better when your team wants a YAML asset in Git and a scheduler around it.
Here is the practical split.
| Tool | Best fit | What to watch |
|---|---|---|
| Debezium | Open-source log-based CDC into Kafka Connect, Debezium Server, or streaming infrastructure | You operate the runtime, connectors, offsets, schema changes, and downstream sinks |
| Fivetran | Managed database replication into warehouses with less infrastructure work | Cost, vendor control, sync frequency, and connector-specific behaviour |
| Airbyte | Open-source or managed connector platform with CDC support for common databases | Airbyte's CDC model is scheduled sync, not an infinite stream, so log retention and sync cadence matter |
| Estuary Flow | Streaming-first CDC and materialization for low-latency data movement | Another streaming platform to operate, govern, and connect to the rest of your stack |
| AWS DMS | Database migration and ongoing replication inside AWS | AWS explicitly notes that DMS CDC is not real-time replication and latency depends on workload, network, replication instance, and target capacity |
| Google Datastream | Serverless CDC on Google Cloud, especially into BigQuery and Cloud Storage | Strong fit for GCP-centred stacks, less portable if your CDC architecture needs to stay cloud-neutral |
| Oracle GoldenGate | Enterprise replication, Oracle-heavy estates, and heterogeneous database environments | Powerful, but usually more enterprise-ops heavy than a simple warehouse ingestion job |
| Qlik Replicate | Enterprise CDC and replication across many source and target systems | Similar tradeoff: broad enterprise capability, but with platform cost and operational ownership |
| Bruin + ingestr | Code-first ingestion assets, CDC, transformations, checks, lineage, and governance in one workflow | Best when CDC should be part of a reviewed data pipeline, not a disconnected sync job |
Honestly, the right choice is less about the CDC acronym and more about the operating model.
If Kafka is already the centre of the architecture, Debezium is a natural fit. If the team wants a managed warehouse connector and does not want to run capture infrastructure, Fivetran or Airbyte may be the simpler path. If the project is a cloud migration, AWS DMS or Datastream can be the right boring answer. If the estate is Oracle-heavy or enterprise replication is the main job, GoldenGate or Qlik Replicate deserve a look.
And if the question is "how do we put CDC inside the same repo where we define transformations, tests, owners, metadata, and lineage?", that is where Bruin fits.
Bruin's CDC support lives in the ingestr asset type. In other words, CDC is not configured as a one-off side process that nobody can find later. It is an asset in the pipeline, alongside SQL assets, Python assets, checks, ownership metadata, and downstream lineage.
A simple PostgreSQL CDC asset looks like this:
name: raw.orders
type: ingestr
connection: bigquery
parameters:
source_connection: pg_prod
source_table: public.orders
destination: bigquery
cdc: "true"
cdc_mode: stream
The Bruin docs describe CDC support for PostgreSQL, MySQL/MariaDB, Vitess, and PlanetScale sources through ingestr assets:
wal_level=logical, a publication, and a replication slot.log_bin=ON, binlog_format=ROW, and binlog_row_image=FULL.psdbconnect API over TLS.Under the hood, ingestr's CDC docs expose the lower-level source schemes and runtime controls. For example, PostgreSQL CDC can use postgres+cdc://, MySQL CDC can use mysql+cdc://, and PlanetScale CDC can use ps_mysql+cdc://. ingestr also has streaming flush controls like --flush-interval and --flush-records, and stream monitoring metrics for replication lag and durable row counts.
Bruin adds the pipeline layer around that:
merge semantics for CDC, so updates and deletes can be applied by primary key.That's the main difference. A CDC tool can move changes. A governed CDC pipeline also tells you who owns the data, which downstream models depend on it, what checks ran, what changed in Git, and whether the tables are trustworthy enough for dashboards or an AI data analyst.
CDC has a cost. Not always a vendor cost, but definitely an operational cost.
You may not need CDC when:
CDC can make a messy pipeline faster, but it will not make it governed.
If there is no ownership, no tests, no lineage, no clear schema-change policy, and no destination contract, CDC just moves the chaos more quickly. This is where teams get disappointed: they add a powerful capture layer, then discover their warehouse models still cannot explain what "active customer" means.
The failure modes are not exotic. They are normal data engineering problems with sharper edges.
Source tables change. Columns are added, renamed, removed, widened, narrowed, or repurposed. CDC pipelines need a schema-change policy:
Without this, one application migration can break ingestion, transformations, dashboards, and AI answers at the same time.
Deletes are the reason many timestamp-based incremental jobs lie.
If a row is physically deleted from the source, a normal updated_at query will not see it. CDC can capture delete events, but the destination still needs to decide what to do with them:
That choice is a business and governance decision, not just an engineering one.
Most CDC pipelines start with an initial snapshot, then switch to ongoing changes.
That handoff is critical. If the snapshot and log stream are not coordinated, you can miss changes that happen during the snapshot or apply duplicates after it.
Good CDC tools handle this carefully. Homegrown scripts often do not, mostly because the happy path looks deceptively simple.
CDC events need to be applied in a way that produces a correct destination state even if a worker retries. That usually means designing around primary keys, source positions, transaction boundaries, and idempotent merges.
If "run it again" creates duplicates, the pipeline is not finished.
Database logs are not kept forever. If a CDC process stops for too long and the source removes the old log segment, the pipeline may no longer be able to resume from its saved offset. Then you are looking at a re-snapshot or some careful repair work.
This is why monitoring matters. CDC needs lag alerts, not just job-success alerts.
A practical CDC pipeline usually has these layers:
source database
-> CDC capture
-> raw change events
-> current-state tables
-> transformations
-> checks
-> catalog, lineage, dashboards, AI analyst
The capture layer is only the beginning.
For analytics and AI workflows, the downstream context is just as important:
This is the part Bruin cares a lot about. The open-source side gives you local, code-first pipeline work with Bruin CLI and ingestion with ingestr. Bruin Cloud adds orchestration, catalog, lineage, checks, access controls, SSO, audit logs, and observability around those workflows.
CDC is strongest when it is not a disconnected sync job. It should be visible inside the same workflow where transformations, checks, and downstream data products are reviewed.
Before choosing CDC, ask:
If those questions feel annoying, that's the point. CDC is not hard because the acronym is hard. It is hard because it touches database internals, pipeline reliability, schema contracts, and business meaning at the same time.
CDC means change data capture. It is the process of identifying inserts, updates, and deletes in a source system and moving those changes to another system, usually a warehouse, lakehouse, stream, or operational database.
ETL is the broader process of extracting, transforming, and loading data. CDC is a capture strategy inside that process. Instead of extracting a whole table every time, CDC captures only the rows that changed.
No. Replication usually tries to keep another database copy close to the source shape. CDC emits a stream or table of change events that downstream systems can transform, audit, replay, or merge into different models.
Use CDC when full refreshes are too slow, source tables are large, downstream data needs to stay fresh, or deletes and updates matter. It is especially useful for database-to-warehouse ingestion and event-driven data pipelines.
Log-based CDC reads the database transaction log, such as the PostgreSQL logical replication stream, MySQL binary log, or SQL Server transaction log, instead of repeatedly querying source tables for changes.
Popular CDC tools include Debezium, Fivetran, Airbyte, Estuary Flow, AWS Database Migration Service, Google Datastream, Oracle GoldenGate, Qlik Replicate, and Bruin with ingestr. The right choice depends on whether you want open-source streaming CDC, managed ELT, cloud migration replication, enterprise database replication, or CDC as part of a governed pipeline.
Yes. Bruin supports CDC through ingestr assets. Bruin docs describe CDC support for PostgreSQL, MySQL/MariaDB, Vitess, and PlanetScale sources using cdc: "true". Bruin keeps CDC in the same asset graph as transformations, checks, ownership metadata, catalog, lineage, and Cloud observability.
No. Kafka is common in CDC architectures, especially with Debezium, but it is not mandatory. CDC changes can flow into warehouses, object storage, queues, streams, or another database depending on the tool and destination.
Yes, proper CDC can capture deletes if the source and capture method support them. The destination still needs a clear policy for applying those deletes, preserving them in history, or marking rows as deleted.
No. CDC can be near real-time, but it can also be consumed in micro-batches or scheduled jobs. The important part is that the pipeline reads changes from a reliable change source instead of repeatedly extracting the whole table.
CDC is a better question to ask of operational data:
What changed, in what order, and how do we safely apply it downstream?
Once you think about it that way, the rest of the design gets clearer. You need a source of changes, a stored position, an idempotent destination write, a schema policy, monitoring, and enough governance that people can trust the tables built from those changes.
The capture part is technical. The trust part is the real work.
How Bruin CLI, Bruin MCP, Bruin Cloud, and agent skills can build and maintain a Salesforce to Snowflake ELT pipeline across bronze, silver, and gold layers.
Most AI data analysts live in Slack or a browser. Bruin runs in WhatsApp too. Here is why field, sales, and ops teams prefer asking their data questions there, what it takes to make it actually work, and how to roll it out safely.
Can you just use ChatGPT, Claude, or a coding agent like Codex to analyze your company data? Here is the honest difference between a general AI model and a purpose-built AI data analyst, why a model alone is not enough, and what it takes to get trustworthy answers from live company data.