Guide
Data migration with ingestr
Plan and run a careful source-to-destination migration with ingestr, from the first connection test to the final cutover and gap backfill.
What
This guide walks through a production-shaped data migration without assuming one database, warehouse, or cloud provider. You will install ingestr, configure a small isolated test, choose between full load, incremental loading, and CDC, then write down the evidence and decisions needed for a final cutover.
The examples use placeholders such as $SOURCE_URI and $DEST_URI. Keep credentials in environment variables, a secret manager, or Bruin connections. Do not paste passwords into a shell history, a committed asset, or an agent conversation.
How
The work has three stages:
- Set up the tools and prove that the source and destination can communicate.
- Run a mock migration against an isolated target and tune the configuration.
- Run the approved migration, reconcile the last changes, and keep a tested rollback path until the destination is trusted.
The short version of the tool split is simple. ingestr moves data between supported sources and destinations. Bruin CLI runs ingestr assets as part of a dependency graph and can place SQL transformations, Python code, quality checks, validation, and backfills around them. You can use ingestr by itself, or use it as the ingestion layer in a Bruin project.
Before you start
- A terminal on macOS, Linux, or Windows with permission to install a CLI
- Read access to the source and write access to an isolated destination schema
- A migration owner who can approve the target, validation rules, cutover window, and rollback plan
- A backup and retention plan for the source and destination before the final run
Guide overview
- 1Set up and test ingestrInstall ingestr, configure safe source and destination access, and run a small isolated copy.
- 2Run a mock migrationTune full load, incremental, and CDC settings with representative data before planning the cutover.
- 3Run the final migrationBack up the systems, close data gaps, cut over deliberately, and monitor the destination after the switch.
Frequently asked questions
What is the difference between a full load, incremental loading, and CDC?
A full load copies the selected source state into the destination. Incremental loading copies rows identified by a cursor such as updated_at or an ID. Change data capture reads a database log or change feed so inserts, updates, and deletes can be replayed. The right choice depends on freshness, delete semantics, source support, and acceptable operational complexity.Does ingestr provide a universal checkpoint for every migration?
No. CDC connectors persist resume positions in the destination staging namespace, while ordinary batch runs depend on explicit intervals, idempotent write strategies, and the surrounding scheduler. Bruin also has asset retries and resumable partitioned backfills, which are separate from ingestr CDC offsets.What is the difference between ingestr and Bruin CLI?
ingestr is the open-source movement engine that reads from a source and writes to a destination. Bruin CLI is the broader pipeline framework. A Bruin project can run ingestr assets alongside SQL and Python assets, dependencies, quality checks, validation, and backfills.
Resources
Get help & contribute