A conceptual guide to self-healing data pipelines: what they are and are not, why incident response is mostly evidence gathering that agents are good at, the detect-diagnose-fix-verify-learn loop, the checks, lineage, and editable definitions that make it possible, and the guardrails that keep it safe.
Arsalan Noorafkan
Developer Advocate
Quick answer: a self-healing data pipeline detects a failure, diagnoses the cause, proposes the smallest safe fix, tests it outside production, and either opens a pull request or applies it - then writes the lesson back so the same problem does not return. The important part is what it is not: self-healing does not mean an agent silently changing production. The pipeline does not heal itself. An agent with access to the pipeline does the work, inside limits you set, and the whole thing works only because most incident response is evidence gathering, which agents are genuinely good at.
This is a conceptual post: what self-healing means, why it works, what a platform needs to make it possible, and the best practices that keep it safe. If you want the hands-on build - project setup, context files, running the loop locally or in the cloud - that lives in the Build a Self-Healing Pipeline Agent guide, and I will point to it rather than repeat it here.
The author works at Bruin, and I use Bruin as the running example because it is what I work in every day. The concepts are tool-agnostic. Corrections welcome at [email protected].
Start with the anti-definition, because it is the part people get wrong:
Self-healing should not mean "an agent silently changes production."
A self-healing pipeline is a pipeline whose failures are diagnosed and repaired by an agent that can read the pipeline, run parts of it in a safe place, and propose changes, within boundaries you define. The canonical shape of the loop is:
detect → diagnose → propose → test in dev → open a PR or ask for approval → verify → learn
Notice that "apply the fix" is not the headline. The headline is propose and verify. A self-healing agent that never touched production directly, and only ever handed you a tested pull request with a diagnosis attached, would already be doing almost all of the work. That framing - boring in the right way - is the one to hold onto.
Be careful with the word "self." A pipeline is code and configuration; it cannot fix itself. What heals it is an agent with enough context to understand what "correct" means, enough access to gather evidence, and enough guardrails to know where to stop.
Here is the argument for why this is agent-shaped work at all.
A pipeline fails overnight. Someone opens the run, finds the asset that broke, reads the log, queries the table, works out whether the numbers downstream are wrong, and decides what to do. Most of that is evidence gathering, and evidence gathering is something an agent is genuinely good at. It is reading, tracing, and cross-referencing - not judgement calls.
The judgement call - what to actually change, and whether to change it in production - is the small part at the end, and it is exactly the part you can keep for a human. Self-healing is valuable long before it is autonomous, because it collapses the forty-five minutes of tracing into something that is already done by the time you open the alert.
There is a second-order reason it matters. Agents are increasingly asked to answer questions on top of your data through a context or semantic layer, and a context layer is only as good as the data underneath it. If the pipelines are broken or the data is stale, the context is wrong and the agents reach the wrong conclusions - the classic garbage-in, garbage-out problem. Self-healing keeps the data healthy so everything built on top of it stays trustworthy.
Two comparisons clear up most of the confusion.
Against retries. For a transient failure - a rate limit, a dropped connection, a warehouse that was briefly unavailable - a retry with backoff is the correct and cheapest answer, and you do not need an agent for it. The limit of a retry is that it repeats the same action blindly; it cannot tell a rate limit from a renamed column. Self-healing is different because it finds out what broke first, then acts on the cause. Retries are a stage inside the loop, not a replacement for it.
Against classic alerting. An alert firing is not the first rung of autonomy - you already have that, and it is what everyone is drowning in. Self-healing goes a step further: the agent comes to you not just with an alert, but with the data visualized and analyzed, a likely cause, and often a proposed fix in a pull request ready for review.
| Retry | Alert | Self-healing | |
|---|---|---|---|
| Triggered by | Any failure | A failure or threshold | A failure or an anomaly |
| Output | Runs the step again | "Something broke" | Diagnosis + tested fix + PR |
| Handles cause? | No | No | Yes |
| Gets smarter? | No | No | Writes the lesson back |
This is the platform question, and the answer is specific. A self-healing pipeline detects a problem (schema drift, freshness anomaly, quality failure), proposes a fix (backfill, type cast, mapping rule), and either applies it or files a PR. Doing that reliably requires three things in the platform:
not_null, unique, accepted_values, and range assertions are not just tests - they are the contract for "good data" that both humans and agents read. accepted_values is literally how an agent knows a renamed status is wrong rather than new.The sharp version of this: tools where lineage and quality are external bolt-ons cannot self-heal without integrating multiple systems. Built-in quality checks and column-level lineage are what make self-healing possible without an external observability stack. This is also why asset-centric platforms (define the tables you produce) fit better than task-centric ones (define the steps) - the asset model gives agents the structure they need for lineage-aware fixes. The shorthand for an agent-ready platform is: small in surface area, declarative where possible, and aware of the data it produces.
Everything above depends on context, and context has a failure mode of its own: conflict. If two files describe the same thing differently - one README says revenue excludes refunds, a column description implies it includes them, a glossary entry disagrees with both - the agent has no way to know which is right. It will pick one, and which one is not predictable. Conflicting context does not just weaken an answer; it makes behaviour non-deterministic, and a self-healing agent acting on the wrong definition can "fix" something that was already correct.
So a single source of truth for each fact is not a nice-to-have, it is a guardrail. The rule is one canonical place per definition, and everything else references it rather than restating it. In Bruin, a shared glossary acts as the semantic layer - domains, entities, and attributes defined once - and columns extend those definitions instead of copying them, so order_id or commerce means one thing across every pipeline. That is what stops the same field being called two things in two places.
Two practices keep it that way:
AGENTS.md at the repo root is the natural home for that map: it tells the agent, and the next engineer, where each kind of context lives and which copy is authoritative. The agent reads it before opening any SQL.For a single team, the repo is the single source of truth and this stays simple. In a bigger or more complex organization, context spans many repos and teams, and the same conflict problem reappears at a larger scale - "team" in one system is team_tag in the next, and no single repo owns the shared definitions. There are two honest options:
Either way the principle is unchanged: name the single source of truth, make everything else point at it, and audit that the pointing stays true. An agent can only be trusted to heal a pipeline if it can trust the context describing it.
Self-healing is not one trick. In practice it is a set of rule-based, scope-restricted tasks where the agent can automatically address common issues, or flag, triage, and quarantine the rest for human review. The recurring pattern is a check with a pre-defined action and an alert target, declared next to the asset. A few common classes:
select * silently changes shape - the fix is to make the select explicit. A column type changes - cast it, or infer the new type by querying. A column is renamed - match it by similarity and refactor the references.Two adjacent methods are worth naming because they turn self-healing from reactive to designed:
The important design point: each of these is implemented as a skill - repo-local instructions for how your team wants the agent to behave for a class of work. Skills are not magic and not emergent behaviour. They are closer to "here is our playbook," written down, versioned, and reviewed. That distinction is what makes the behaviour predictable.
The single most useful framing: how much of the loop the agent runs on its own is a setting, not a law. Some teams want a diagnosis and nothing else. Some want the whole loop to run unattended overnight. The dial is governed by permissions, instructions, and your risk appetite - and it does not have to be one setting for the whole system.
The rule that scales well is to escalate by blast radius. The more a change can affect, the more oversight it earns:
Two habits keep this honest. First, the agent should always propose the smallest safe fix - correct the specific check that a genuine upstream rename broke, not silence a class of failures. Second, containment: a fix should rerun the narrowest affected asset set, bounded by asset and date range, never a blind full rebuild. A backfill can rewrite months of history, so treat wide backfills and anything that writes past the warehouse as needing a human.
The failure mode to design against is confident wrong answers. The countermeasure is grounding: make the agent read the logs and the lineage and show which rows it looked at, rather than reason from what it half-remembers. And where the agent is unsure, the right behaviour is to ask a question rather than guess - the same principle that keeps the context side honest.
Most of what makes self-healing safe is unglamorous and worth stating plainly.
The goal is not "never wrong." It is that when the agent is wrong, you find out quickly and can see why. Everything above - the repo, the pull requests, the checks, the named scope - exists so that a mistake is visible and reversible. That is a far lower bar than perfection, and it is the achievable one.
The stage teams skip is the last one, and it is what separates a loop from a one-off script. When an incident is resolved, the correction has to be written back somewhere durable - a check, a description, a glossary entry, a data contract - so the same gap cannot open again silently.
Writing it into the agent's memory is not enough; that is private to one session and unversioned. Writing it into the repo means it is reviewed, it survives, and the next engineer and the next agent both start from it. Every incident should leave behind a check, a description, or a rule. If it leaves behind nothing, you will have it again. Agents are relentless in a way people are not - if you instruct one to always follow the same process and update the docs, it does, with no "oh, I forgot" - which is exactly why the write-back is worth automating.
If you want to build this rather than reason about it, the Build a Self-Healing Pipeline Agent guide walks through it end to end: importing your tables as assets, enriching their definitions, setting the scope and permissions, and running the loop locally or as a shared cloud agent. The concepts on this page are the "why"; that guide is the "how."
A self-healing data pipeline is one whose failures are diagnosed and repaired by an AI agent with access to the pipeline, within limits you set. The agent detects a problem, diagnoses the cause, proposes the smallest safe fix, tests it outside production, and opens a pull request or asks for approval. It does not mean an agent silently changing production. The pipeline does not repair itself; an agent with the right context and guardrails does.
No. A retry repeats the same action blindly and works for transient failures like a rate limit or dropped connection. A self-healing pipeline first works out what actually broke. Most of that work is evidence gathering - reading run logs, following lineage, querying tables - which is something agents are genuinely good at. It can tell a rate limit from a renamed column and correct the cause rather than rerun the failing step.
Three things in the platform: declared quality checks at the asset level so the agent knows what good data looks like, column-level lineage so it can scope the blast radius of a change, and a definition format the agent can safely read and edit. Where lineage and quality are external bolt-ons, a pipeline cannot self-heal without integrating multiple systems.
The agent cannot tell which is right, so it picks one unpredictably. Conflicting context makes behaviour non-deterministic, and a self-healing agent can end up "fixing" something that was already correct. Prevent it with a single source of truth for each definition: one canonical place per fact (a shared glossary for definitions, the README for intent, the asset file for columns), everything else references it, an AGENTS.md that maps where context lives and which copy is authoritative, and regular audits for duplication and drift. Beyond one repo, either own each shared definition in exactly one place across repos, or use a dedicated context or semantic layer as the canonical directory.
Classic alerting tells you something broke. Self-healing goes further: the agent arrives with the incident already investigated - the data analyzed, the likely cause identified, and a proposed fix in a pull request ready for review. Most observability tools detect and some diagnose, but they sit outside your repo, so they cannot fix, verify, or write the lesson back.
Autonomy is a setting, not a law. It ranges from diagnosis-only to running the whole loop unattended, and where you sit is governed by permissions, instructions, and risk appetite. A good rule is to escalate by blast radius: low-risk, reversible changes can auto-open a pull request, while anything touching finance metrics, definitions, or write-back fields should ask for approval.
Give the agent read-only production access plus a non-production database it can break, use separate least-privilege credentials for pipelines and agents, keep the loop as a reviewed pull request rather than a silent production edit, log every write, treat declared checks as the contract for good data, implement each behaviour as a repo-local skill rather than emergent magic, and roll self-healing out last, after the reporting layer is trusted.
A practical 2026 shortlist of the best data pipeline tools, including a direct Dagster vs Prefect answer for teams choosing between assets and Python flows.
A simple guide to ingesting Salesforce data into Snowflake, modelling bronze, silver, and gold layers, and using Bruin agents for dashboards, Slack reports, activation, and self-healing pipelines.
A practical framework for building an AI context layer using open-source tools, turning skepticism about AI in data engineering into a working solution with self-healing pipelines and iterative team adoption.
Practical updates on open-source data pipelines, AI analysts, governance, and what we are shipping at Bruin.
The signup form is hosted by Brevo. Accept cookies to load it.