Stripe Analytics Pipeline with BigQuery
1) Run the raw layer
Bruin's selector syntax lets you run one directory of assets at a time:
bruin run --selector "path:assets/stripe_raw" \
--start-date 2023-01-01 --end-date $(date -u +%F) \
--full-refresh --no-validation
You run the three layers one at a time rather than in a single bruin run. Nothing requires it - the ingestion step is just the slow one, because it is the only step that waits on the Stripe API, and a problem with a key or a permission is cheaper to find here than after a full DAG run. Once the pipeline works, the daily command is a plain bruin run.
Three things matter on this first load:
--start-dateand--end-dateset the window the run covers. This is what decides how much history you get, and it is the easiest thing here to miss.--full-refreshbuilds the tables from scratch. Use it on the initial load only.--no-validationskips the query dry-run. Validation covers every asset in the pipeline even when you select one directory, and on a new BigQuery destination the downstream tables do not exist yet, so it would fail for the wrong reason.
Warning
--start-date is not optional here. The raw assets use ingestr's incremental Stripe mode, which only fetches objects created inside the run's window, and bruin run defaults that window to yesterday. Without it your first load is one day of Stripe data and every report downstream is empty. Set it to the earliest history you actually need.
Warning
Do not keep --full-refresh in your daily schedule. On the raw layer it replaces each table with whatever the run's window returned instead of merging into it, and on the staging layer it discards the accumulated daily MRR snapshots - observations that cannot be reconstructed from current Stripe data.