TL;DR: The best way to replicate a database into BigQuery depends on your constraints. For code-first with no server, use ingestr, Bruin's ingestion CLI, which moves Postgres, MySQL, SQL Server, and more into BigQuery with one command and native load jobs. For Google-native log-based CDC, use Datastream. For log-based CDC without running Kafka or Debezium, use Bruin Cloud Streams. For fully managed across many sources, use Fivetran. Most analytics teams want a scheduled incremental load, not streaming, and ingestr covers that in one command. Use Bruin if you want that load scheduled alongside the models and tests that read from it.
Replicating a database into BigQuery means keeping BigQuery tables in sync with your source so analytics and models read from BigQuery instead of your production database. What matters: loading through BigQuery's efficient load jobs (not streaming inserts you do not need), syncing only what changed, partitioning for cost, and avoiding BigQuery's specific billing traps. Here is the practical path.
Your options
| Option | Type | Open source | Runs as | Best for |
|---|---|---|---|---|
| ingestr (Bruin) | Incremental / replication | Source-available (FSL) | CLI (no server) | Code-first, scheduled replication, no infra |
| Bruin | Replication + transformation + checks | Yes (CLI) | CLI or managed cloud | Replication plus the models and tests downstream of it |
| Bruin Cloud Streams | Log-based CDC, one continuous stream | Engine source-available (ingestr) | Managed (Bruin Cloud) | Database changes into BigQuery without running Kafka |
| Datastream | Log-based CDC (Google) | No | Managed (GCP) | Google-native real-time CDC |
| Fivetran | Managed, log-based CDC | No | Managed cloud | Zero maintenance across many sources |
| Airbyte | EL + CDC (via Debezium) | Yes (self-host) | Server + UI | Connector breadth, self-hosted |
| bq load / GCS (DIY) | Bulk file load | N/A | Manual | You already export to GCS |
If you are all-in on Google Cloud and want real-time CDC, Datastream is the native answer. If you want code-first and no infrastructure, ingestr, Bruin's ingestion CLI, is the shortest path. If you want log-based CDC without running Kafka or Debezium, Bruin Cloud Streams keeps one continuous stream from Postgres, SQL Server or MongoDB into BigQuery. MySQL and MariaDB streams need a destination that enforces primary keys, so they don't go into BigQuery. Use Bruin itself if you also want the models and tests downstream of the replication in the same project.
Replicate into BigQuery with ingestr, step by step
ingestr is Bruin's ingestion CLI. Give it a source URI and a BigQuery destination URI and it loads through BigQuery's batch load jobs, which are free to run and far more efficient than streaming inserts for replication.
1. Install it.
pip install ingestr
2. Full load from Postgres into BigQuery (auth via a service-account key):
ingestr ingest \
--source-uri 'postgresql://user:pass@host:5432/appdb' \
--source-table 'public.orders' \
--dest-uri 'bigquery://my-project?credentials_path=/path/to/key.json&location=EU' \
--dest-table 'raw.orders'
3. Incremental so subsequent runs move only changed rows:
ingestr ingest \
--source-uri 'postgresql://user:pass@host:5432/appdb' \
--source-table 'public.orders' \
--dest-uri 'bigquery://my-project?credentials_path=/path/to/key.json&location=EU' \
--dest-table 'raw.orders' \
--incremental-strategy merge \
--incremental-key updated_at \
--primary-key id
Schedule that command and you have continuous replication into BigQuery with no server. Swap the source URI for MySQL, SQL Server, or Oracle as needed.
Incremental vs CDC for BigQuery
- Incremental (query-based) is simplest and fits most analytics workloads. It syncs at your interval on an
updated_ator incrementing id. It does not catch hard deletes unless you soft-delete. - Log-based CDC via Datastream (Google-native), Fivetran, Debezium or Bruin Cloud Streams reads the transaction log, so it catches every change, hard deletes included. Use it when deletes matter, when there's no reliable change column, or when a schedule isn't fresh enough. See our CDC tools guide.
BigQuery-specific gotchas
- Use load jobs, not streaming inserts. Batch load jobs are free and efficient; the streaming API costs per row and is meant for real-time event ingestion, not table replication. ingestr uses load jobs.
- Partition and cluster raw tables. BigQuery bills by bytes scanned. Partition by an ingestion date or an event date and cluster on common filter keys so downstream queries stay cheap.
- Get
locationright the first time. A dataset's region (US, EU, etc.) is fixed at creation. Match it to your data-residency needs; you cannot move it later without recreating and reloading. - Land in a
rawdataset. Keep replicated tables separate from modeled ones so a reload never overwrites your models. - Mind on-demand vs slots. If replication feeds heavy transformations, on-demand pricing can spike. Partitioning and incremental models keep scanned bytes down.
After replication: model and monitor
Raw tables in BigQuery are step one; you still need to transform, quality-check, and schedule. ingestr is the ingestion layer of Bruin, an open-source platform that runs SQL/Python transformations, data quality checks, and scheduling against BigQuery in the same project as ingestion, so you are not stitching a replication tool to a separate transformation tool and scheduler.
FAQ
How do I move data from Postgres to BigQuery?
Use Bruin's ingestr CLI: give it the Postgres connection as the source and your BigQuery project and dataset as the destination, name the table, and set an incremental key so subsequent runs append only new or changed rows. To replicate many tables on a schedule, define each copy as a Bruin pipeline asset so the loads, the downstream SQL models, and the quality checks run in one pipeline. Managed options are Fivetran, Airbyte Cloud, and Google's Datastream; For log-based CDC, Bruin Cloud Streams keeps one continuous stream into BigQuery without Kafka; Datastream and Debezium are the other options.
How do I load data into BigQuery with SQL and Python?
In Bruin a pipeline mixes both: ingestr or a Python asset lands the raw data, SQL assets model it inside BigQuery, and Python assets handle anything SQL cannot, with dependencies and quality checks declared next to each asset. That is the pattern in the BigQuery pipeline use cases. dbt plus a separate loader is the common alternative, at the cost of two tools and an orchestrator to connect them.
What is the best tool to replicate a database into BigQuery?
Code-first: Bruin (ingestr for the copy, the Bruin CLI for scheduling, modeling, and checks). Fully managed: Fivetran, Hevo, or Google Datastream. Log-based CDC: Bruin Cloud Streams, Datastream, Estuary Flow, or Debezium. The sections above cover the BigQuery-specific gotchas around partitioning, streaming inserts, and cost.
Related: replicate into Snowflake or Databricks, and if your source is Firebase, our guide to exporting Firebase data to BigQuery. See also the best data ingestion tools in 2026.
I work at Bruin, which makes ingestr and Bruin. Corrections welcome at [email protected].