Data Quality and Testing Strategies for Modern Data Pipelines

TL;DR: Data quality in a modern pipeline is two jobs. The first is a gate: checks that run with the pipeline and stop bad data before it lands downstream. The second is a monitor: something that watches what still got through. The open-source frameworks split the same way. Bruin declares checks on the column inside the asset, so they run on every bruin run and block by default; Great Expectations and Soda Core run as a separate validation step; dbt tests cover a dbt-only transformation layer; Elementary, Monte Carlo, and Anomalo watch after the fact. This guide covers how to choose between them, how to add checks for nulls, duplicates, ranges, freshness, and completeness, and how to turn those checks into a data contract that producers and consumers can both hold each other to.

Most data quality programmes fail for a dull reason: the checks live somewhere other than the code that produces the data. A validation suite in one repository, a transformation in another, a monitoring dashboard in a third. The three drift apart within a quarter, and the incident that finally gets attention is the one no check covered. The strategy below is built around keeping the check next to the thing it protects.

Evaluating open-source and enterprise data quality tools

The tool landscape in 2026 sorts into two kinds, and the useful question is not "which has more check types" but "where does the check run".

Best data quality tool by need:

• Checks that run with the pipeline and block bad data by default: Bruin
• Widest library of ready-made expectations: Great Expectations
• Readable YAML checks and data contracts as a separate step: Soda Core
• Tests inside an existing dbt project: dbt tests
• Anomaly monitoring on a dbt project without a licence: Elementary
• Managed observability across a large warehouse: Monte Carlo or Anomalo

Three trade-offs decide the choice.

Build versus buy. The open-source gates cost engineering time to wire in and nothing to run. The managed monitors cost a licence and very little time, but they tell you about bad data after it landed. Teams under budget pressure should spend on the gate first, because a blocking check prevents the incident that an observability tool would only report.

Unified versus bolted on. A separate validation suite is a separate thing to keep in sync. Every renamed column, every deleted model, every new table is a change in two places. Tools that put the check in the asset definition remove that class of drift. This is the reason Bruin, which is our product, declares checks on the column inside the SQL or Python file that produces it: rename the column and the check moves with it, delete the asset and the check goes with it.

Coverage versus rules. Rule-based gates cover what you thought to write a rule for. Anomaly monitors cover what you did not, at the cost of false positives and a bill. Most teams that get past a few dozen tables end up with both: one gate, one monitor, reading from the same definitions.

Implementing automated data quality checks

The community line "every AI project starts with good data" is true, and the practical version of it is that validation belongs at the point of ingestion and transformation, not in a report the next morning. Three categories of check catch most incidents.

Integrity: keys, nulls, and duplicates

The first checks on any table are the ones that make joins safe. In Bruin they are declared in the asset header, in the same file as the SQL that produces the table:

/* @bruin
name: mart.orders
type: sf.sql
depends: [raw.orders, raw.customers]
materialization:
  type: table
columns:
  - name: order_id
    type: integer
    primary_key: true
    checks:
      - name: not_null
      - name: unique
  - name: customer_id
    type: integer
    foreign_key:
      table: mart.customers
      column: customer_id
    checks:
      - name: not_null
      - name: relationships
  - name: status
    type: string
    checks:
      - name: accepted_values
        value: [placed, paid, shipped, refunded]
  - name: order_total
    type: float
    checks:
      - name: non_negative
      - name: max
        value: 100000
@bruin */

SELECT o.order_id, o.customer_id, o.status, o.order_total, o.updated_at
FROM raw.orders o

bruin run executes the checks as part of the asset run. Every check is blocking by default, so a failure stops the downstream assets rather than letting a broken table feed a dashboard. A check you want to observe without stopping the pipeline gets blocking: false.

The built-in column checks cover the common cases: not_null, unique, positive, non_negative, negative, accepted_values, pattern for a regular expression, min and max for thresholds on numbers and dates, and relationships for referential integrity against another asset's column. The equivalent in Great Expectations is an expectation suite with expect_column_values_to_not_be_null and friends, run from a checkpoint; in Soda it is a checks for orders: block with missing_count(order_id) = 0 and duplicate_count(order_id) = 0; in dbt it is not_null, unique, accepted_values, and relationships tests in the model's YAML.

Freshness and completeness

The check that catches the most expensive incidents is the one that notices a load did not happen. Two custom checks cover it. In Bruin, custom_checks take any SQL that returns a value to compare:

/* @bruin
name: mart.orders
type: sf.sql
depends: [raw.orders]
custom_checks:
  - name: loaded in the last 6 hours
    query: SELECT max(updated_at) > current_timestamp - interval '6 hours' FROM mart.orders
    value: 1
  - name: today has rows
    query: SELECT count(*) FROM mart.orders WHERE updated_at::date = current_date
    count: 1
    blocking: false
@bruin */

The first is a freshness check: the newest row must be recent. The same two checks belong on the raw ingestr asset that lands the data, so a stalled load is caught at the source. The second is a completeness check on today's partition, marked non-blocking so a slow source produces a warning rather than a stopped pipeline. Soda expresses the same idea as freshness(updated_at) < 6h and row_count > 0; Great Expectations as expect_column_max_to_be_between on the timestamp; dbt through the source freshness command on sources.

Continuous monitoring is the same checks on a schedule. Because Bruin runs the checks with the asset, a pipeline that runs every hour re-validates freshness and completeness every hour without a separate monitoring job. The gap this leaves is the unknown unknown, a table whose distribution shifts in a way no rule anticipated, and that is the job for an anomaly monitor.

Where to run the checks

Run them twice. On every pull request, run the pipeline against a development target and let a failing check block the merge. In production, run them with every scheduled run and route failures to the channel the team watches. The CI half is covered in running data pipelines in CI/CD with GitHub Actions; the short version for Bruin is bruin validate on the pull request and bruin run on merge.

Enforcing data contracts in pipelines

A data contract is a formal agreement between the team that produces a table and the teams that consume it: these columns, these types, these guarantees, and a promise that changes are announced before they ship. Most write-ups treat it as a document. It only works as code.

The asset definition above is already a contract. The columns block names the fields, their types, and the checks the producer promises will hold. What turns it from a habit into a contract is enforcement in two places.

At build time, policies. Bruin reads a policy.yml at the project root and applies its rules on bruin validate and automatically before every bruin run:

rulesets:
  - name: contracts
    rules:
      - asset-has-owner
      - asset-has-description
      - asset-has-columns
      - asset-has-primary-key
      - asset-has-checks
      - pipeline-has-notifications

An asset that skips the column list, has no primary key, or carries no checks fails validation before it can run. Custom rules use boolean expressions over the asset metadata, so a team can require, for example, that every asset in the mart schema has a not_null check on its primary key. Soda's data contracts and dbt's model contracts (contract: enforced: true with declared types) play the same role in their ecosystems.

At run time, the checks. A contract is only real if breaking it has a consequence. Blocking checks provide it: the producer's own pipeline stops when the promised guarantee fails, before a consumer sees the bad rows. The relationships check is the contract between two assets, since it fails when a foreign key points at a row that does not exist in the referenced table.

Impact before the change ships. The half of the contract about announcing changes is where lineage earns its place. If the pipeline framework knows which downstream assets read a column, a pull request that removes or retypes it can fail validation with the list of what breaks. Bruin derives that graph from the asset definitions and the SQL it parses, and bruin lineage mart.orders --full prints every upstream and downstream dependency. Pair it with the checks and the contract covers both halves: what the data must look like, and who has to be told when it changes.

How to choose

• A small team with one pipeline framework: put the checks in the assets, make them blocking, add a policy file that requires them, and run bruin validate in CI. Bruin does all four from one CLI; dbt tests plus a build gate is the equivalent for a dbt-only team.
• A platform team with many producers: add Great Expectations or Soda as a shared validation layer with a central library of expectations, and require producers to publish contracts.
• Hundreds of tables and a budget: keep the gate, add a monitor. Elementary if you run dbt, Monte Carlo or Anomalo if you want coverage without writing rules.

Whatever the stack, the principle is the same: the check lives where the data is produced, it blocks by default, and it runs in CI and in production from the same definition. Everything else in data quality follows from that. For the tool-by-tool comparison see the best data quality tools in 2026, and for the lineage half of the contract, the best data lineage and catalog tools.