New in Bruin Cloud
Cloud Streams: CDC database replication
CDC used to take Kafka, Debezium and a week. Now it's 1 click.
Every insert, update and delete, from your database to your warehouse.
Productionpublic.orders| id | customer | status | total |
|---|---|---|---|
| 10461 | Parcel Pilot | shipped | 310.00 |
| 10470 | Northwind | cancelled | 120.00 |
| 10477 | Kestrel Health | paid | 89.50 |
| 10482 | Vela AI | paid | 240.00 |
Cloud Streamsreading the WAL
- insert104820/1A3F2B8
- update104770/1A3F4C0
- delete104700/1A3F5E8
continuous
Warehouseanalytics.orders| id | customer | status | _cdc_deleted | _cdc_synced_at |
|---|---|---|---|---|
| 10461 | Parcel Pilot | shipped | false | 13:52:04 |
| 10470 | Northwind | cancelled | false | 14:01:12 |
| 10477 | Kestrel Health | paid | false | 14:05:40 |
| 10482 | Vela AI | paid | false |
Trusted by forward-thinking teams
How it works
Turn it on once. It keeps up from there.
Once
Turn on your database's change log.
Logical replication on Postgres, a row-based binlog on MySQL, CDC on SQL Server.
1 click
Start the stream in Bruin Cloud.
Bruin copies each table once, then follows the log.
Continuous
Every change streams into your warehouse.
One stream that keeps running. Not scheduled batches.
Compared
CDC without the Kafka cluster.
| Question | Kafka + Debezium, built yourself | Fivetran or Airbyte | Bruin Cloud Streams |
|---|---|---|---|
| What you run | Kafka, Kafka Connect, Debezium and a sink connector | Nothing | Nothing |
| Setup | Days to weeks | A setup wizard per source | 1 click, once the change log is on |
| How changes arrive | A continuous stream | Scheduled batch syncs. Fivetran's 1-minute syncs need its Enterprise plan | One continuous stream |
| Run the engine yourself | Yes | Airbyte: yes. Fivetran: no | Yes, with ingestr |
What it handles
The parts that usually break, handled.
$ ingestr ingest --source-uri postgres+cdc://… --stream
- snapshot · public.orders1.2M rows
- streaming · WALmerge on id
- connection lost14:02:11
- resumed · 0/1A3F2B814:02:40
Picked up where it stopped. Nothing skipped.
Deletes
Kept and flagged with _cdc_deleted, or removed. Your call.
New columns
A column added in Postgres reaches the warehouse without a restart.
Your tables
Stream the whole database, or only the tables you name.
Full refresh
Rebuild a table from a fresh copy while the stream keeps running.
Lag you can see
Bruin Cloud tracks how far behind each stream is, and alerts you.
Sources
Seven databases, read from their own logs.
Into BigQuery, Snowflake, StarRocks, PostgreSQL, DuckDB and more.
Same engine
Runs on ingestr. Run it yourself, too.
Cloud Streams runs ingestr's CDC. The same stream works from your laptop or CI with one command.
# The same stream, from your own machine
ingestr ingest \
--source-uri 'postgres+cdc://user:***@host:5432/shop' \
--dest-uri 'bigquery://my-project?credentials_path=sa.json' \
--streamThe platform
Part of the Bruin platform.
Streams land as raw tables, ready for your SQL models, quality checks and lineage.
Sources
DatabasesWarehousesApps & APIsFiles & storageStreams & webhooksWeb scrapingMove
Data IngestionFrequently asked
Questions about Cloud Streams.
What is change data capture (CDC)?
CDC copies every insert, update and delete from a database's own change log, instead of re-querying its tables. Bruin Cloud Streams uses it to keep your warehouse in sync with production.
Do I need Kafka or Debezium to use Bruin Cloud Streams?
No. Bruin reads the database log directly and merges each change into your warehouse, so there is no Kafka cluster or connector to run.
What do I need to set up on my database before streaming with Bruin?
Turn on the change log once: logical replication and a replication user on Postgres, a row-based binlog on MySQL, CDC or Change Tracking on SQL Server, a replica set or Atlas cluster on MongoDB. On managed databases such as AWS RDS this is a parameter change that can need a restart.
Which databases and warehouses does Bruin Cloud Streams support?
It reads PostgreSQL, MySQL, MariaDB, SQL Server, MongoDB, Vitess and PlanetScale from their own change logs, and writes to BigQuery, Snowflake, StarRocks, PostgreSQL, MySQL, SQL Server and DuckDB, among others. MySQL and MariaDB streams need a destination that enforces primary keys, so not BigQuery or Snowflake.
Does Bruin Cloud Streams slow down my production database?
The first sync copies each table once. After that, Bruin reads the database's change log instead of querying your tables again.
How fresh is the data that Bruin streams into my warehouse?
Continuously. Bruin Cloud Streams is a stream, not a scheduled batch load, so changes keep flowing into your warehouse instead of waiting for the next sync.
How does Bruin Cloud Streams handle deleted rows?
By default a deleted row stays in the warehouse with _cdc_deleted set to true, so you keep its last values. Switch to hard deletes and the row is removed.
What happens if a Bruin stream stops or the warehouse is down?
The stream resumes from the last change it saved, so nothing is skipped. Delivery is at-least-once and changes are merged by primary key, so on tables with a primary key a replay doesn't create duplicate rows.
Will a stalled Bruin stream fill up my Postgres disk with WAL?
A Postgres replication slot keeps WAL until the stream confirms it, so a stalled stream does make the log grow. Bruin Cloud tracks how far behind each stream is, in WAL bytes for Postgres, and alerts you when it falls behind.
Can Bruin replicate tables without a primary key?
Bruin mirrors tables by primary key. On Postgres, a keyless table with REPLICA IDENTITY FULL is captured as an append-only change log instead; other databases skip keyless tables with a warning.
What happens when someone adds or changes a column in Postgres?
A running stream picks up the change and updates the warehouse table without a restart. Dropped columns stay in the warehouse as empty columns.
How is Bruin Cloud Streams different from Fivetran or Airbyte?
Fivetran and Airbyte copy changes in scheduled batches, and Fivetran's 1-minute syncs need its Enterprise plan. Cloud Streams is one continuous stream, on ingestr, an engine you can also run yourself.
Can I run the same CDC without Bruin Cloud?
Yes. ingestr ingest --stream runs the same stream on your own machine or servers; ingestr is source-available under the Functional Source License. Bruin Cloud keeps it running, tracks lag and alerts you.
Your warehouse, in sync with production.
A demo walks through your own data.