2026 W28

I am super excited to have announce Change Data Capture (CDC) support in ingestr.

CDC allows streaming changes to the database in near real-time. It allows replicating data between databases with very little latency, and allowing incremental replication even for cases where there is no cursor column to replicate from. Historically, CDC required quite a bit of infrastructure to be built and implemented, including Kafka clusters, Debezium connectors and whatnot.

We have decided to take a stab at it: ingestr already supports tens of different database systems, has very robust connector capabilities, and many destinations it already supported. The challenge has been implementing stream management into a batch solution.

ingestr now supports a new --stream flag that allows long-running processes to be replicated.

Bruin CLI

New Features

• MongoDB and MongoDB Atlas support: bruin query runs finds and aggregations against MongoDB and MongoDB Atlas using a JSON envelope, bruin data-diff compares MongoDB collections using document-inferred schemas, and bruin import database pulls MongoDB collections in with optional --as-ingestr and --destination flags.
• SQL unit testing: The new bruin unit-test command runs SQL asset logic against fixture data by rewriting refs to CTEs. It supports inline and shared fixtures, partial and exact matching, CTE-level assertions, and freezing execution_time. Works across the major SQL platforms.
• Standalone Python scripts: bruin run can execute a standalone Python script without any pipeline scaffolding.
• Three-part table names: Asset names can now use the full database.schema.table form across all supported engines.
• Dremio and Sail platforms: Both are now first-class platforms with dedicated dialects and schema/folder auto-creation, replacing the generic flight.sql platform.
• Fabric everywhere: bruin data-diff supports Microsoft Fabric, Fabric schemas are created automatically instead of erroring, and Fabric formatting defaults to the TSQL dialect.
• Cloud objects from the CLI: New bruin cloud dashboards and bruin cloud agents commands to list, get, create, and update dashboards, agents, and agent prompts, plus bruin cloud connections add/list/delete that read from your local .bruin.yml and upload service account file contents automatically. bruin cloud trigger run gained downstream-trigger options.
• Configurable retries: Set retries at the asset, check, or pipeline level and Bruin will retry failed steps according to that policy.
• Structured asset parameters: Asset parameters accept nested maps and lists, not just flat key/value strings.
• Variant control: Templated (variant) assets support an enabled flag so you can skip specific variants.
• Snowflake per-asset warehouse override: Individual assets can override the warehouse size; a per-warehouse client cache reuses connections across assets that share the same override.
• Ingestr materialization block: Ingestr assets can declare a materialization config block; column overrides accept sized strings such as varchar(50); individual columns can be masked with mask flags; and any additional ingestr flag can be passed through as an asset parameter.
• Streaming Python generators: materialize() writes dict, list-of-dict, and Arrow yields to disk in batches, so large Python assets no longer OOM.
• Column-metadata modeling: DDL generation now captures foreign keys, defaults, precision, scale, length, and collation across the major warehouses. A new valid-column-metadata lint rule catches misuse.
• AWS default credential chain for secrets: Environment variables, IAM roles, and instance profiles are all accepted, alongside the new global --secrets-backend flag.
• Secrets masked in run output: Credential values are redacted in bruin run output and log files, including ingestr subprocess output. --debug also forwards to the ingestr subprocess so you can see its debug logs.
• Nested variables in query: bruin query --var supports dot notation and JSON values, so you can override deeply nested inputs from the CLI.
• Skills init: bruin skills init scaffolds AI skills, including a ready-to-run self-heal demo.
• Salesforce OAuth: Salesforce connections accept an OAuth access_token in place of username, password, and security token.
• Reddit Ads OAuth: Reddit Ads connections use OAuth refresh tokens that auto-renew.
• Oracle Data Integrator importer: A new bruin import variant pulls in assets from ODI.

Improvements

• Full-refresh restrictions honored: Assets marked as not allowed to full-refresh now fall back to incremental with a warning during pipeline runs rather than erroring out.
• Dataproc Serverless log streaming: Driver logs stream directly from the GCS staging bucket instead of Cloud Logging.
• Resilient validate: bruin validate <asset> degrades gracefully when a sibling pipeline is broken and resolves cross-pipeline external:// dependencies against the whole repo.
• Interactive Python progress bars: Carriage-return output from tqdm and similar tools is preserved in the terminal, and is kept out of the log file.
• Ingestr credential validation: Cassandra, Granola, and Reddit Ads credentials are validated before any asset runs.

Bug Fixes

• Windows: asset discovery in bruin format and credential paths on Windows are fixed.
• MSSQL, Fabric, and Synapse: cross-database dev-schema rewrites, T-SQL surrogate key varchar casts, and Fabric column metadata fetching are fixed. create-replace honors typed columns; data-diff count parsing on Snowflake is corrected.
• Ingestr: --columns preserves declared integer widths, routes naive timestamps to timestamp_ntz, and fixes interval and bigdecimal mappings. SendGrid and ESPN URIs include the :// prefix so params are no longer dropped.
• Config: env var interpolation preserves unset variables literally instead of erroring out.
• Query: bruin query --limit no longer breaks ClickHouse functions such as toFloat64.
• Cloud CLI: connections resolve relative service_account_file paths against the config directory; agent update clears fields explicitly for empty flags; dashboard --state validation handles ambiguous flags, empty values, and timestamp-like scalars.
• Vertica DDL emits DEFAULT before NOT NULL.
• Table names with empty components (e.g. raw..events) are rejected instead of producing invalid SQL.
• Salesforce, Tableau, and Vitess schemas relax fields that should not be required, and Vitess grpc_tls no longer forces TLS on the MySQL-protocol port.
• Secrets-backend errors are surfaced to callers instead of being logged as "does not exist".
• Snowflake honors the database component of three-part asset names and accepts bare one-component names for existence checks.
• bruin import errors from Tableau, BigQuery scheduled queries, and QuickSight are surfaced; Tableau host is normalized.

Breaking Changes

• Athena requires explicit credentials: Athena connections now require either AWS access keys or an AWS profile.
• PlanetScale renamed: The connection type is now planetscale_mysql and the ingestr URI scheme is ps_mysql://.
• Ingestr CDC materialization: Requires the merge strategy. The legacy cdc append strategy and non-incremental materialization keys are rejected.
• Start date on full refresh: bruin run --full-refresh always uses the CLI start_date and no longer falls back to the pipeline's start_date.
• Reddit Ads account selection: The account_ids field on the connection was removed. Select accounts via a source-table suffix such as campaigns:id_123.
• catchup merged: catchup and catchup_mode are now a single catchup field accepting a bool or "active"/"all".
• Dremio flat names: Dremio assets must use folder.table names.
• flight.sql platform removed: Replaced by the dedicated dremio and sail platforms.

Bruin Cloud

Dashboards

• Dashboards as Code: DAC dashboards render in the dashboards UI with a full YAML viewer, a read-only widget inspector, row tabs, and full widget and chart parity with UI-created dashboards. DAC dashboards can be deleted from the UI.
• Fork a dashboard: Duplicate an existing dashboard from the index page with one click.
• Sortable dashboard table with a none-state on the index.
• Chart type swap between bar, line, and area from the widget inspector.
• Edit widget titles and axes: A pencil button opens editors for axis titles, y-axis number format, and widget rename.
• Widget width picker: A manual width picker replaces the drag-to-resize UI.
• Live SQL streaming preview on the empty canvas while the agent generates, alongside streaming agent reasoning and prose.
• Full-screen query inspector with expandable command view.
• Semantic layer support in dashboards, with widget SQL scoped to the current viewer.
• Attach widgets to chat with an explicit paperclip button.
• Export chat threads as JSON from dashboards.
• Browser notifications when the dashboard agent finishes.
• Dashboards can be created without a connection, and the new-dashboard modal shows the selected agent's description and active connections.

AI Agents and Chat

• Live agent activity stream with a detail panel; agent messages surface in the Activity panel.
• Multi-repo agents: Agents can span multiple repositories via a unified repository picker.
• Scheduled runs from chat: Draft scheduled runs are rendered inline as cards in chat threads, with an in-conversation edit flow.
• Edit or create an agent from the chat's agent dropdown.
• Bruin remembers your last-used agent per user on the chat page.
• Continue typing while the agent is running, with a stop-running button.
• Reload banner when the underlying prompt is edited during an active chat.
• Reasoning acknowledgment notes appear while the sandbox is booting so you know why the agent is not responding yet.
• Inline connection prompts: When the agent needs a connection to answer, an "add connection" block appears directly in web and dashboard chats. A specific nudge is shown when Databricks is missing.
• Team-level model selection and codex reasoning effort, with Opus 4.8 available.
• AI disclaimer below the chat input.
• Team-scoped skills in AI Agent Settings.
• Attachment upload failures are surfaced instead of being silently dropped; attachments are also processed inside scheduled-run planning chats.

Runs and Pipelines

• Asset Health tab: A new tab on the asset detail page shows a timeline of runs with drill-down, colorblind-friendly patterns, and a clear message when a schedule is unsupported.
• Manual health entries: Add, edit, and delete manual entries (formerly "placeholder runs") on the Asset Health timeline, with inclusive end dates, notes, prefilled dates, and toast confirmation.
• External Dependencies section on the asset instance page with inline slot bars, coverage summary, drill-down menu, manual entry per upstream, and grouped layout.
• Run tag filtering across the runs UI, including backfill trigger tags.
• Priority range extended to -10..10 with a Select dropdown.
• GitHub link button on the asset file path.
• Symbolic upstreams shown as dashed edges in the lineage graph with a tooltip.
• External upstream dependencies are visible in the run lineage graph before a run starts and on the run detail page.
• Assets that have not started yet appear in the run lineage graph.
• Rerun controls disabled while a run is unfinished; rerun correctly clears downstream statuses.
• Backfill single-interval routes through the single-run flow for consistency.
• Per-attempt asset version and a run-version (commit) column are shown in the asset runs table; "Go to Asset" opens at the clicked run's version.
• Run variables card: OXR variables and pipeline variable states are surfaced; the card has copy buttons and improved layout.
• Collapsible asset descriptions with a "Show more" button.

Scheduled Runs

• Multi-channel Slack: Post scheduled-run notifications to multiple Slack channels, and post directly to a channel.
• Per-schedule timezone with UTC shown in the dropdown when empty.
• Cross-run memory: Opt-in cross-run state files that can be edited from the UI and rendered as readable markdown.
• Live-validated Slack channels at setup, with channel autocomplete and name validation, paginated picker, and add-by-ID with name resolution.
• Thread link in Slack alerts and a "view conversation" link under the Slack progress spinner.
• Back link to the list from the scheduled run page.
• Agent result email with an inline chart rendering, delivered to a configurable address.

Cost Explorer

• Drill in from the timeline: Click a bar in "Cost Over Time" to drill into that period.
• Multi-select pipeline and asset filters, with asset filter scoped to the selected pipeline.
• Avg runtime column and a runtime dot-strip with p50/p95/p99 tooltip in the query analysis table.
• Expand "Other" groups on click to drill into them in the table.
• Refresh button and a "This month" quick range for paygo usage.

Billing

• Monthly message limit banner: Dismissible and collapsing to a pill, enforced on both manual and scheduled-run paths, with a per-team monthly cap on paygo.
• Team Stripe invoices are listed in billing settings.

Public API

• Connections: POST /api/v1/connections to create a connection.
• Agents: Create, read, update endpoints, plus read/write for the agent system prompt.
• Dashboards: Read-only endpoints for lists and details, plus POST /api/v1/dashboards to create dashboards into draft.
• Semantic layer: Agent API with read and write-back.
• Pipeline metadata: A run metadata update endpoint.

Integrations

• Databricks per-user OAuth: New connection type with a derived redirect URL, copyable callback URL, and a warning when connection sets reference a deleted app.
• WhatsApp: Disconnect from the UI, plus file attachments on deferred resend.
• Update hosted project dataset and table selection from project settings.

Improvements and Bug Fixes

• Allow disabling a pipeline even when its connections are missing.
• Runs export: fix zero-row rows and drop bogus asset-count columns; restrict export to the authenticated user's email.
• Asset logs: fix download order, add copy-all, and add in-page search.
• Chart rendering fixes for negative values and crowded x-axis labels.
• Long links no longer overflow chat bubbles.
• Home chat composer: send button centered and textarea scroll fixed.
• Run details: fix missing external assets, unknown-type badges, brief RUNNING flashes, and stuck UNKNOWN backfills.
• Live asset-check refreshes and better retry and failed-check handling on run details.
• Dashboard chat scheduled-run cards no longer persist across refresh.
• Runs list "Load more" flicker on filter fetches fixed.
• Email verification banner no longer covers the layout.
• Snowflake test-connection route fixed.
• Onboarding: "Invite team" and "Chat integrations" get real completion signals; "Add integration" deep-links into agent Slack setup.
• StarRocks placed in the database group in the connection dropdown; connection type list sorted alphabetically within each group.

Breaking Changes

• PlanetScale renamed from planetscale to planetscale_mysql.
• Fabric and OneLake: The use_azure_default_credential option was dropped.
• Password length capped at 72 characters.
• Email changes disabled on the profile endpoint.
• Email verification required with a grace period.

Bruin VS Code

• Local backfill with interval chunking: A new "Backfill…" entry in the Run dropdown splits a date range into hourly, daily, weekly, or monthly chunks and runs bruin run sequentially per chunk. Includes a granularity selector (defaulted from the asset schedule), a stop-on-failure toggle, and a live chunk preview. Per-chunk runs are grouped under a single expandable "Backfill" row in the Run History panel with a rolled-up status, and each row shows its window.
• Multi-column sort in query preview: Click column headers to sort by several columns at once; the last click is primary and previous clicks become tie-breakers. A "Sorted by" bar shows active columns with controls to flip direction, remove a column, or clear all.
• Pipeline-wide column-level lineage: Opening "Column Level Lineage" from the pipeline view now renders every asset with all asset- and column-level edges. Assets that participate in column lineage open with columns expanded so column-to-column arrows are visible.
• Tableau assets and 50+ new asset types: The YAML schema and completions cover 107 asset types, up from 54. Tableau assets no longer trigger validation errors and get autocompletion, a badge, and lineage-node styling.
• Run configuration popover: The "Run with" chip strip is replaced by a small info icon in the dates icon group. A dot indicates that non-default options are active. Clicking opens a popover listing active flags, tag filters, and variable overrides; each row jumps to its editor.
• Config schema for all connection types: config-schema.json now covers all 136 connection types from the CLI (previously only environment was defined). A scheduled workflow keeps it in sync. Athena and other profile-based connections no longer lint-error.
• Human-readable cron labels in the schedule CodeLens, matching the cloud app, and weekly scheduling aligned to Sunday to match the cloud scheduler.
• Lineage panel performance: The lineage webview is retained when hidden, the Vue app and elkjs graph are no longer rebuilt on every editor switch, document edits are debounced, and switching files while the panel is hidden keeps its contents current.
• Selecting many assets to run no longer produces a corrupted terminal command; runs launch via a temporary script file.
• No install-screen flash on startup for users who already have the CLI installed.
• Column Level Lineage view no longer resets to pipeline view on every data refresh; it only resets when you change asset or pipeline.
• Expand button on lineage nodes now zooms out so newly added upstream or downstream nodes are visible.
• Query preview truncation warnings and pending sorts are scoped per tab, cleared when a tab closes, and dropped when a tab reruns its query.

ingestr

New Sources

• New streaming sources: Azure Event Hubs, NATS, Apache Pulsar, Redis Streams, Google Pub/Sub, AWS SQS, MQTT topics, plus expanded Kinesis.
• New CDC sources: SQL Server Change Tracking (via mssql+ct, sqlserver+ct, azuresql+ct, azure-sql+ct), MySQL and MariaDB CDC (mysql+cdc, mysql+pymysql+cdc, mariadb+cdc), MongoDB CDC (mongodb+cdc) with snapshot, resume, batch, and stream modes, and Vitess/PlanetScale CDC via VStream and psdbconnect with dedicated vitess:// and ps_mysql:// schemes.
• New database sources: native Db2 via DRDA, StarRocks with lakehouse catalog support.
• New SaaS sources: Amplitude, Google Search Console, GitLab, Trello, Square, Recurly, Chargebee, Braze, Twilio, Reddit Ads, Paddle, SendGrid, SharePoint, Monday items, ESPN, football-data.org, API-Football, and BallDontLie FIFA.

New Destinations

• New destinations: Apache Iceberg with REST and SQL catalogs, Oracle with a thin driver, StarRocks via Stream Load, Google Sheets, and Vitess/PlanetScale as write destinations through the MySQL destination.
• Iceberg supports the full range of write strategies: merge, delete+insert, truncate+insert, and SCD2.

New Flags and Features

• --stream: Continuous streaming ingestion with --flush-interval (default 30s) and --flush-records (default 50,000). Works with postgres+cdc, mssql+cdc, kafka, and amqp, with at-least-once delivery via merge.
• --progress json: Structured line-delimited JSON output for CI, jq, and orchestrators.
• _ingestr_loaded_at column: A load-timestamp column is added to loaded tables by default. Disable with --no-load-timestamp.
• --trim-whitespace: Trim string values during ingestion, also available as a Python option.
• Extract partitioning: Time and integer partition columns with auto-sized windows for bounded reads.
• Python SDK: pip install ingestr[sdk] provides a Python SDK with Arrow-stream and mmap transports (Python 3.8+).
• Column type coverage: tinyint/int8 and sized string types such as VARCHAR(255) are accepted in --columns overrides and propagated across destinations.
• Three-component table names: catalog.schema.table works across BigQuery, Snowflake, DuckDB, and MSSQL sources and destinations.
• AWS MSK OAUTHBEARER: Kafka source authenticates against MSK.
• HubSpot service keys are accepted as credentials.

Improvements

• Postgres CDC performance: Parallel flush and snapshot, binary transport, and protocol v2 streaming deliver roughly 30% faster catch-up at about half the client CPU. Schema changes are handled automatically, new eligible tables are added to the managed publication, replication lag is exposed via HTTP expvar, and unchanged TOAST columns are filled from prior rows in the decoder.
• BigQuery merge cost reduction: New tables are clustered on up to four primary key columns by default, cutting scanned bytes by roughly 84% on key-local deltas. Merges prune by partition when the partition field is part of the merge key. Schema-inferred sources use pre-staged JSONL loads at extract time.
• MySQL destination: Standard MySQL and MariaDB writes now use LOAD DATA LOCAL INFILE; chunked parallel INSERT is retained for Vitess and PlanetScale.
• CSV to DuckDB: Parallel parsing, conversion, and appends while preserving output order.
• MSSQL destination: Writes optimized and empty-target merges improved.
• Fabric destination: New write_strategy parameter defaulting to copy (TDS bulk copy) instead of parameterized INSERT.
• OneLake destination: Accepts period-separated Delta table targets and skips Fabric-managed area directories.
• SQL Server source: Bracketed N-part identifiers and catalog-qualified tables supported.
• CDC merge semantics: Uniform end-to-end across DuckDB, Postgres, BigQuery, Snowflake, SQLite, MySQL, MSSQL, ClickHouse, CrateDB, MongoDB, DynamoDB, and Cassandra.
• Snowflake decimals: Metadata is preserved on schema introspection, so reruns no longer collapse NUMBER(20,2) to decimal(38,0).
• Braze: New tables including user_data, event_series, campaign_series, canvas_series, sessions, purchase_quantity, purchase_revenue, and segment_series. user_data exports all segments by default.
• SendGrid: Batched message extraction, richer suppression, blocks, spam, and unsubscribe tables, half-open interval semantics.
• Reddit Ads: OAuth refresh-token auth and account auto-discovery.
• Monday: Rate-limit and complexity budget handling with Retry-After and retry_in_seconds; api_token accepted as an api_key alias.
• SharePoint: User-Agent decorated to reduce throttling.
• Google authentication: Application Default Credentials supported.
• Salesforce: History tables (e.g. casehistory) added for existing objects.
• Adjust: attribution_types configurable for campaigns and creatives.
• Stripe: 429 rate-limits retried with exponential backoff.
• Telemetry is non-blocking so CLI startup is not held up by network. Debug logs use zap-compatible ISO8601 tab-separated format; column override logs print readable type names.

Bug Fixes

• CDC: Postgres slot lag no longer grows during idle streaming; walsender stays alive during destination-write phase; keepalive is sent immediately on start; array columns are preserved; UUID copy encoding is fixed. _cdc_unchanged_cols is kept out of user merges.
• BigQuery: Duplicate query jobs are recovered when jobs.insert returns 409; load-job batches are cast to the staging table schema; the known-datasets map is protected from concurrent writes.
• Snowflake: Multi-table write deadlock from connection pool exhaustion is fixed; parquet timestamp loading is fixed.
• Databricks: JSON array decoding for ARRAY<INT> results from statement results is fixed.
• ClickHouse: Array and low-cardinality array ingestion are fixed.
• DuckDB: Replace staging uses the target schema; CDC LSN result is cloned to survive Arrow buffer release.
• MSSQL: Bulk copy on TIME(n) columns is fixed; custom-query decimal scale is preserved; UUID formatting on Snowflake targets is fixed. MSSQL and MySQL read queries use original source column names.
• Postgres: UUID copy encoding, primary key logs.
• Delete+insert transaction handling is corrected across BigQuery and other destinations.
• No-op string schema migrations are avoided when Adjust decimal metrics feed unbounded string columns.
• Indeed traffic_stats preserves per-job-and-platform rows.
• BigQuery merge no longer ALTERs on int-PK narrowing on reruns.
• Schema evolution Arrow ownership and mmap Arrow slice timestamp corruption are fixed.

Breaking Changes

• PlanetScale scheme renamed from planetscale to ps_mysql. Vitess and PlanetScale now use dedicated schemes.
• Python SDK requires Python 3.8+ and pyarrow >= 14.0.1 for a CVE fix.