What is CDC streaming? Learn how it reads database logs, where Kafka fits, how it differs from batch CDC, and when its operational cost is worth it.
Arsalan Noorafkan
Developer Advocate
Quick answer: CDC streaming is change data capture delivered as a stream. Instead of waiting for a nightly job to copy changed rows, a CDC reader watches the source database log and emits inserts, updates, and deletes as they happen, or close enough that downstream systems behave like they are live.
The simple version:
database transaction log -> CDC reader -> stream or topic -> consumers and destinations
CDC streaming is what you reach for when "sync the table every hour" stops being good enough.
Sometimes that is because product analytics needs fresh events. Sometimes it is because a warehouse table is too large for repeated incremental queries. Sometimes it is because multiple systems need to react to the same customer, order, payment, or inventory change. And sometimes it is because a team has already learned the hard way that polling updated_at misses deletes.
For the broader capture method, start with what is change data capture. This guide focuses on the continuous, low-latency delivery model.
The names change between PostgreSQL, MySQL, SQL Server, Oracle, Kafka, Kinesis, Pub/Sub, Debezium, Fivetran, Airbyte, Estuary, and Bruin, but the design questions stay pretty consistent: what changes are captured, how are they ordered, where do they go, how do you replay them, and what happens when the pipeline falls behind?
CDC stands for change data capture. It means detecting changes in a source system and making them available somewhere else.
CDC streaming is the streaming version of that idea.
A database table changes:
Then a CDC process reads those committed changes and sends them downstream as events.
A CDC streaming event usually contains:
A simplified CDC event might look like this:
{
"source": {
"database": "app",
"schema": "public",
"table": "orders",
"position": "16/B374D848"
},
"operation": "update",
"key": {
"order_id": 1042
},
"before": {
"status": "pending"
},
"after": {
"status": "paid"
},
"committed_at": "2026-07-13T09:42:12Z"
}
That position is not decoration. It is how the CDC system knows where it is in the source log. Without a durable offset, restart behaviour becomes guesswork.
Most data pipelines start as batch jobs because batch jobs are easy to reason about. Run every hour. Pull changed rows. Merge into the destination. Done.
That works for a lot of tables.
It starts to break when the table is large, mutable, and operationally important.
The classic polling pattern is:
select *
from orders
where updated_at > :last_run_time
It is simple, but it has weak spots:
CDC streaming moves the capture point closer to the database itself. The database already records changes in a log so it can commit transactions, recover after failures, and replicate state. CDC uses that log as the source of truth.
The result is usually lower latency and a cleaner change history.
That does not mean CDC streaming is free. It adds more moving parts. But when freshness and correctness matter, those moving parts can be worth it.
Every implementation has its own details, but the shape is usually the same.
An application changes data. The database commits the transaction and records the change in an internal log.
Common examples:
The important part: the database log is ordered, durable, and tied to committed transactions.
The CDC reader connects to the source as a replication consumer or equivalent. It reads changes in order and translates database-specific log records into a more useful change event.
This reader might be an open-source connector, a managed ELT connector, a cloud migration service, or an ingestion tool embedded inside a broader pipeline platform.
The reader must remember the last source position it processed safely.
Depending on the source, that offset might be:
Offset handling is where a lot of reliability lives. A CDC streaming job should be able to stop, restart, and continue from the last safe point without silently skipping committed changes.
The next hop depends on the architecture.
Common patterns:
CDC streaming does not always mean Kafka. Kafka is common because it is good at buffering, retaining, partitioning, and replaying streams, but the CDC concept is broader than one platform.
Downstream consumers do different jobs:
This is why CDC streaming is not just "replication." Replication usually aims to keep another copy close to the source. CDC streaming often creates a reusable stream of business changes that multiple consumers can interpret differently.
Both models capture changes. The difference is delivery.
| Area | Batch CDC | CDC streaming |
|---|---|---|
| Runtime | Runs on a schedule | Long-running or near-continuous |
| Latency | Minutes to hours | Seconds to minutes |
| Failure mode | Retry the next batch | Resume from a durable offset |
| Destination writes | Larger batches | Small batches or continuous writes |
| Operational concern | Job duration, batch windows | lag, backpressure, offset health |
| Best fit | Analytics syncs that can wait | live analytics, event-driven systems, operational replicas |
Batch CDC is still useful. If a finance table updates a few times per day, a scheduled CDC job may be enough. Streaming CDC is useful when delay matters or when multiple consumers need a live change feed.
The mistake is treating "real time" as automatically better. Real time costs you something: infrastructure, monitoring, schema contracts, backpressure handling, and on-call expectations.
CDC streaming usually sits between operational systems and analytical or event-driven systems.
Typical sources:
Typical destinations:
Typical use cases:
In a modern data stack, CDC streaming is often the ingestion layer. It does not replace transformation, quality checks, lineage, access control, semantic models, or dashboards. It makes fresher raw changes available so the rest of the stack has better inputs.
Use CDC streaming when at least one of these is true:
Good examples:
created, paid, fulfilled, and refundedCDC streaming is especially useful when one source change has several downstream meanings. An order update might refresh a warehouse table, trigger a fulfilment workflow, update a fraud model, and appear in a customer support tool. A stream lets those consumers move independently.
Do not use CDC streaming just because it sounds more modern.
It may be the wrong default when:
Sometimes the boring answer is correct. A daily snapshot for a tiny reference table is fine. An append-only event export may be simpler than CDC. A timestamp cursor may be enough if deletes are soft-deleted and updates are rare.
CDC streaming earns its keep when it solves a real latency, correctness, scale, or fan-out problem.
CDC streaming starts at the source database, so source setup matters.
For PostgreSQL, this usually means logical replication, a publication, and a replication slot. For MySQL, it usually means binary logging with row format and enough retention. Other databases have their own equivalent settings.
Two practical questions matter early:
If log retention is too short, a broken consumer can fall behind past the point of recovery. At that point you may need a new snapshot.
CDC streams capture changes from a point in time. Most pipelines also need the current contents of the table before streaming begins.
That bootstrap is the initial snapshot.
A good setup defines:
The handoff between snapshot and stream is one of the most important correctness points in the whole design.
Most CDC events should be keyed by the source primary key. That lets downstream systems merge changes into the correct row and lets streaming platforms partition related events together.
Ordering is more subtle.
You usually need ordering per row or per primary key. Global ordering across every table is expensive and often unnecessary. For Kafka, key choice affects partitioning, and partitioning affects ordering. If all events for order_id = 1042 go to the same partition, consumers can process that order's changes in sequence.
Deletes are one of the reasons teams adopt CDC in the first place.
Decide how deletes are represented:
operation = deleteThe destination behaviour should be explicit. Some analytics tables should remove deleted rows. Some should preserve a delete event for audit. Some should keep the last known row and mark it inactive.
Source schemas change. Columns are added, renamed, widened, narrowed, or removed.
CDC streaming makes schema changes more visible because the stream is always moving. If the destination schema or consumer code cannot handle a new event shape, the pipeline can fail quickly.
Plan for:
People love saying "exactly once." In practice, the safer design habit is this: assume retries happen, and make the destination idempotent.
That usually means:
CDC streaming can be very reliable, but only if replay is boring.
CDC streaming systems fail slowly before they fail loudly. Lag grows, queues fill, log retention shrinks, and the source slot starts retaining too much history.
Watch:
Lag is not automatically bad. A short spike during a bulk update is normal. Persistent lag means the destination or consumer cannot keep up.
CDC readers often need elevated source permissions. They may see columns that normal application users do not query directly.
Treat CDC streams as production data:
This is where CDC streaming crosses from "connector setup" into data platform design.
Kafka is common in CDC streaming because it gives you a durable middle layer.
With Kafka, a source table often maps to one or more topics. CDC events are written to those topics. Consumers read independently using consumer groups. Kafka keeps offsets, stores events for a retention period, and lets consumers replay from earlier positions if needed.
Kafka helps when:
Kafka also adds work:
So the rule is simple: Kafka is useful when you need a streaming backbone, not when you only need one database table copied into one warehouse every hour.
Imagine a Postgres orders table.
The application writes orders all day:
A polling job might run every 15 minutes and query rows where updated_at changed. That can work, but it leaves gaps: hard deletes, timestamp bugs, and a constant delay.
A CDC streaming setup does this instead:
public.orders.orders_current table.Now the warehouse can stay close to the operational database without rescanning the whole table. And if someone asks why an order changed status three times in ten minutes, the raw CDC events can explain it.
Before you call a CDC streaming pipeline production-ready, answer these questions:
If those answers are vague, the implementation is not ready yet. The connector may run, but the pipeline is not operationally understood.
Bruin supports CDC streaming through ingestr assets. For PostgreSQL, the Bruin PostgreSQL CDC stream mode docs show CDC enabled with cdc: "true" and cdc_mode: stream, using PostgreSQL logical replication with the usual publication and replication slot setup.
Example:
name: public.orders
type: ingestr
connection: bigquery
parameters:
source_connection: my_pg
source_table: public.orders
destination: bigquery
cdc: "true"
cdc_mode: stream
Bruin also supports Kafka as a source for ingestr assets. The Bruin Kafka ingestion docs describe a Kafka connection with bootstrap_servers and group_id, then an asset that reads a topic like kafka.my_topic into a destination.
That means Bruin can cover both sides of the common CDC streaming picture: database CDC through ingestr, and Kafka topic ingestion when Kafka is already part of the streaming architecture. The important bit is that the stream can live in the same reviewed pipeline graph as transformations, checks, lineage, and governance, instead of being a disconnected sync job nobody remembers until it falls behind.
CDC streaming is the continuous movement of database changes from a source system into downstream systems. It captures inserts, updates, and deletes from a database log or equivalent change feed and emits them as events.
CDC is the broader method of capturing source changes. CDC streaming is the low-latency version where changes are delivered as an ongoing stream instead of only being collected during scheduled batch jobs.
Usually CDC streaming is near real time, not instantaneous. Latency depends on source log availability, connector speed, buffering, network, streaming platform, and destination writes. For most data systems, seconds or low minutes are close enough to real time.
No. Kafka is common because it is good at durable streams, replay, partitions, and consumer groups. But a CDC streaming pipeline can also write directly to a warehouse, object storage, a queue, or another streaming service.
ETL is the broader process of extracting, transforming, and loading data. CDC streaming is one way to extract and deliver database changes. Teams still need transformations, quality checks, access controls, documentation, and downstream modelling after the stream exists.
It depends. CDC streaming is useful when freshness, deletes, updates, replay, or fan-out matter. Batch ingestion is often simpler for small tables, low-change tables, or workflows where hourly or daily freshness is enough.
Common tools and platforms include Debezium, Kafka Connect, Estuary Flow, Fivetran, Airbyte, AWS DMS, Google Datastream, Oracle GoldenGate, Qlik Replicate, Kinesis, Pub/Sub, and Bruin with ingestr.
Yes. Bruin supports PostgreSQL CDC stream mode through ingestr assets using cdc: "true" and cdc_mode: stream. Bruin also supports Kafka ingestion through ingestr assets, so Kafka topics can be part of a governed Bruin pipeline.
How Bruin CLI, Bruin MCP, Bruin Cloud, and agent skills can build and maintain a Salesforce to Snowflake ELT pipeline across bronze, silver, and gold layers.
Most AI data analysts live in Slack or a browser. Bruin runs in WhatsApp too. Here is why field, sales, and ops teams prefer asking their data questions there, what it takes to make it actually work, and how to roll it out safely.
Can you just use ChatGPT, Claude, or a coding agent like Codex to analyze your company data? Here is the honest difference between a general AI model and a purpose-built AI data analyst, why a model alone is not enough, and what it takes to get trustworthy answers from live company data.