How do I check null rate on Databricks?
Declare the check on the asset that produces the data, not in a separate suite. In Bruin you add a checks block to the column inside the Databricks asset definition and bruin run executes it as part of every run, failing the pipeline before downstream tables read bad data. Because the check lives in the same file as the SQL, it cannot drift out of sync with the column it protects. Great Expectations and Soda Core do the same job as a separate step, which is the better fit if you need checks across tools Bruin does not produce.
Command
bruin runDefined in
YAML
Works with
Databricks + Bruin CLI
What you get
How to do it
- 1
Open the Databricks asset that produces the table you want to guard.
- 2
Add a checks entry under the relevant column for the null rate rule.
- 3
Set blocking: true so a failure stops the run rather than logging a warning.
- 4
Run bruin validate to confirm the definition parses.
- 5
Run bruin run against Databricks and confirm the check appears in the run output.
- 6
Wire the same command into CI so the check runs before a merge reaches production.
How it works in code
columns:
- name: order_id
checks:
- name: null_rate
blocking: trueRun bruin run and Bruin asserts that the null share of a column stays under a threshold on Databricks before any downstream asset runs.
Worth knowing
A check that only warns will be ignored within a month. Set blocking: true on the checks that matter. On Databricks specifically, an all-purpose cluster left warm for a nightly job costs far more than a job cluster that starts and stops
Other ways to do this
Bruin is not always the right answer. Here is where the alternatives are stronger.
| Option | When it is the better choice |
|---|---|
| Bruin | Verify that a column has not started filling with nulls on Databricks, as a blocking step inside the pipeline rather than a separate monitoring job. |
| Great Expectations | The largest open-source expectation library. Better if you need checks across sources Bruin does not produce, at the cost of running it as its own layer. |
| Soda Core | Readable YAML checks that a stakeholder can review, and the strongest option for formal data contracts between teams. |
| Monte Carlo / Anomalo | Managed observability that learns Databricks baselines automatically. Catches what you did not write a rule for, but alerts after the fact rather than blocking. |
Common questions
How do I check null rate on Databricks?
Add a checks block to the column in the Databricks asset definition and run the pipeline. Bruin asserts that the null share of a column stays under a threshold and fails the run when it does not hold.
Can a null rate check block a Databricks pipeline?
Yes. Set blocking: true and the run stops on failure, so downstream assets never read the bad data. Without it the check logs a warning and the pipeline continues.
What is the best open-source way to check null rate?
Bruin, Great Expectations, and Soda Core are all open source and all handle it. Bruin declares the check inside the asset so it runs on every pipeline run; the other two run as a separate step, which suits checks spanning tools outside your pipeline.
Catch bad data before it ships
Open source. Built-in and custom checks that run on every pipeline run.