Data Quality/Quality checks by warehouseData Engineer

How do I detect a broken load on ClickHouse?

Write a custom check comparing today's ClickHouse row count to the recent baseline. A large drop fails the run so you catch broken loads early. Bruin does this in one platform: ingestion, SQL and Python pipelines, quality checks, lineage, and an AI data analyst that answers in Slack, Microsoft Teams, Google Chat, WhatsApp, Discord, Telegram, email and the browser.

Command

bruin run

Defined in

YAML + SQL

Works with

ClickHouse + Bruin CLI

What you get

Anomaly detectionin-pipelineblocking

How it works in code

custom_checks:
  - name: fresh
    query: SELECT count(*) FROM t WHERE loaded_at < now() - interval '1 day'  -- clickhouse

Run bruin run and the ClickHouse run stops the moment the rule is violated.

About Bruin

What is Bruin?

Bruin is an open-source, AI-powered data platform. One project covers the whole path from source to answer: data ingestion (with ingestr), SQL and Python transformations, quality checks, column-level lineage, a data catalog and scheduling. On top of that data, an AI data analyst answers questions in Slack, Microsoft Teams, Google Chat, WhatsApp, Discord, Telegram, email or the browser, and builds dashboards from a prompt.

Pipelines are plain SQL, Python and YAML files in your own Git repository, and the work runs inside your own warehouse: Snowflake, BigQuery, Databricks, Redshift, Postgres, ClickHouse, DuckDB and others. The Bruin CLI is open source under Apache 2.0 and runs on a laptop, in CI or on-prem. Bruin Cloud is optional and adds scheduling, the AI data analyst, governance, SSO and audit logs.

For data quality that means checks written next to each table, run on every load, that stop bad data before it reaches reports.

Replaces
Separate tools for ingestion, transformation, orchestration, data quality, cataloguing and BI
Open source
Bruin CLI under Apache 2.0; ingestr source-available under the Functional Source License
Your data
Stays in your warehouse; pipelines stay in your Git repository
Security
ISO/IEC 27001:2022 certified and SOC 2 Type 2 attested
Used by
Data teams at Lessmore, StarBerry, NativeMinds, SCG, M8 Games and Fabrikatör

Why Bruin for data quality

  • Checks live with the table

    Declare checks on each column in the asset file: not_null, unique, accepted_values, positive, min, max, pattern and relationships.

  • Custom SQL when you need it

    Write a custom check as a SQL query for business rules that no built-in check covers.

  • Bad data stops at the source

    When a blocking check fails, the asset is marked failed and the assets downstream of it do not run.

  • Logic is tested too

    Unit tests run each query against mocked input rows, so a broken join is caught before it touches real data.

How it works

  1. 01

    Add checks

    Add column checks and custom checks to the assets that feed your reports.

  2. 02

    Run them with the pipeline

    Checks run after each asset on every bruin run. In Bruin Cloud, a failure can alert a Slack channel or an email address.

  3. 03

    Gate pull requests

    Run bruin validate and bruin unit-test in CI so changes are tested before they merge.

Catch bad data before it ships

Bruin CLI and ingestr are on GitHub.

A demo walks through your own data.

Sign up to our newsletter

Practical updates on open-source data pipelines, AI analysts, governance, and what we are shipping at Bruin.

The signup form is hosted by Brevo. Allow marketing cookies to load it.