Incremental vs. Full Refresh Runs in Bruin
Learn how Bruin's interval start/end variables drive incremental runs, how --full-refresh changes the picture, how each behaves across SQL, Python, and ingestr assets, and how to protect critical tables with refresh_restricted.
Run modes
Three different things in Bruin are called start_date - distinguish them before reading the table:
start_date | Where it lives | Role |
|---|---|---|
| Pipeline default | start_date in pipeline.yml | Earliest point full-refresh will rewind to |
| Schedule-derived runtime | Computed by Bruin per scheduled run | Start of the previous schedule interval, set automatically |
| Manual runtime | --start-date CLI flag | Ad-hoc override for backfills |
These three sources map directly to the three ways a run can be triggered:
| Command | Runtime start_date | Runtime end_date | When to use |
|---|---|---|---|
bruin run --full-refresh | pipeline default start_date | now | First run, or full rebuild |
bruin run | schedule-derived (start of previous interval) | schedule-derived (start of current interval) | Regular scheduled runs |
bruin run --start-date X --end-date Y | manual (X) | manual (Y) | Manual backfills of a specific window |
These rules are schedule-agnostic - they apply identically to @hourly, @daily, @weekly, @monthly, or any cron expression. The window simply tracks the schedule cadence one-for-one (hourly schedule → one-hour window, weekly → one-week, and so on).
The classic "load history once, then incrementals" pattern falls straight out of this: trigger the first run with --full-refresh to absorb history from the pipeline default start_date, then let the schedule take over.
Interval modifiers
To catch late-arriving events or add a lookback/lookahead on top of the runtime interval, use interval modifiers - they shift start_date and end_date by a fixed offset. Set them on the asset (overrides) or the pipeline (default):
# Asset-level: shift this asset's window back 2 hours
interval_modifiers:
start: -2h
end: 0h
# Pipeline-level: applies to every asset that doesn't override it
default:
interval_modifiers:
start: -1d
end: 0h
Modifiers only kick in when the run is invoked with --apply-interval-modifiers, so it's an opt-in per run.
More guides
Capstone: add returns to the pipeline
Extend the project, prove the result, and assess the finished pipeline.
Describe the model in its asset definition
Keep the metric definition, owners, limits, and checked examples next to the SQL.
Load and profile the source data
Load local commerce data and check its shape before modeling it.