
Shopify to ClickHouse: build a fast ecommerce analytics warehouse
Learn how to load Shopify data into ClickHouse, choose an ingestion method, model a bronze-silver-gold ecommerce warehouse, and use Bruin to spend less time maintaining data tooling.
Stripe's dashboard, Sigma, and Data Pipeline each stop somewhere. Here is where they stop, how a three-layer Stripe analytics pipeline fixes it, how the tooling options compare, and why starting from a free local template beats assembling a stack.

Arsalan Noorafkan
Developer Advocate

Quick answer: Stripe's dashboard, Sigma, and Data Pipeline each answer a slice of the revenue question and stop. They stop at Stripe's data, Stripe's metric definitions, and Stripe's idea of what history is worth keeping. A Stripe analytics pipeline fixes all three by landing Stripe objects in your warehouse, modelling them with definitions your team owns, and snapshotting the state daily so churn and retention are measurable at all. You do not need to buy anything to start: bruin init stripe-bigquery gives you 19 assets, four report tables, and a dashboard that run locally for free.
The usual trigger is a meeting. Someone shows the Stripe MRR number, someone else shows the number from the board deck, and they differ by four percent. Nobody in the room can say which is right, because neither number has a definition anyone can read.
That is not a Stripe problem. Stripe is doing exactly what a payments platform should do. It is a scope problem, and it shows up in every company that grows past the point where "check the dashboard" is a reporting strategy.
Worth being precise here, because Stripe's tooling is better than the "just use a BI tool" crowd usually admits.
| Surface | What it does well | Where it stops |
|---|---|---|
| Dashboard reports | Payments, payouts, balance, and billing summaries with no setup | Fixed metrics, Stripe data only, no joins to anything else |
| Billing analytics | Subscription counts, MRR, churn, growth trends out of the box | One definition of MRR, no way to change it |
| Sigma | Read-only SQL over your transactional data, inside the Dashboard | Stripe data only, no external BI connection to the underlying store, data available up to a data_load_time boundary rather than live |
| Data Pipeline | A managed data share into Snowflake, Redshift, Databricks, or BigQuery, refreshed on a regular full load | Gives you raw Stripe tables; the modelling, metric definitions, and reports are still yours to build |
| Revenue Recognition | Accrual accounting, ASC 606 style treatments, auditable revenue schedules | Built for accounting, not for product or growth analytics |
Notice the pattern. Every one of these is excellent within Stripe's boundary and silent outside it. Data Pipeline is the closest thing to a solution, and even it hands you raw tables and wishes you luck - which is the honest place to start the real conversation.
Five specific limits, in the order teams usually hit them.
The questions that matter are almost never about billing alone:
Every one needs Stripe joined to something else - product events, your CRM, the support tool, the ads platform. No Stripe surface can do that join, because the other side of it is not in Stripe.
Stripe's MRR is a specific formula. So is your finance team's. They agree until they do not, and the disagreements are always in the same places: annual plans, discounts and coupons, proration on mid-cycle upgrades, metered usage, trials, past-due subscriptions, and multi-currency accounts.
You cannot edit Stripe's formula. You can only build your own alongside it and then spend every quarter explaining the gap. Once the definition lives in a SQL file in your repository, the explanation is a diff.
This is the technical limit that surprises people, and it is the most important one.
Stripe objects are mutable. When a customer downgrades from $99 to $49, the subscription record changes in place. The $99 does not go anywhere - it never existed as a separate historical fact. Query Stripe today and you get today's state, correctly, and no trace of last month's.
Which means: you cannot compute churn, expansion, contraction, or net revenue retention from current Stripe data. Not with better SQL, not with a smarter tool. The information is gone. The only fix is to write down what you observed, each day, and never rewrite it. That is a pipeline, and it has to start running before it can tell you anything.
The corollary is uncomfortable: the best day to start a Stripe pipeline was a year ago.
A metric defined in a BI tool's query builder has no diff, no reviewer, and no blame. When the churn number moves, "did the data change or did the definition change?" becomes an archaeology project.
The same metric as a SQL file in Git has an author, a pull request, a reason, and a test. It is the same metric. The difference is entirely in whether you can defend it six months later.
Every Stripe reporting add-on and every managed connector prices against how much data moves or how many transactions you process. Your reporting needs do not grow at that rate. A company processing ten times the payments does not need ten times the reports - it needs roughly the same six reports, computed correctly.
This mismatch is fine when the reports are worth it. It stings when you are still figuring out whether they are.
Three layers. That is the whole idea, and it has not changed in twenty years of data warehousing.
Raw. Stripe objects, loaded as they came, in a schema of their own. Unix timestamps and nested JSON intact. This layer exists so that when a number looks wrong you can prove what Stripe actually sent. Nobody reports off it.
Staging. Typed, flattened, joined. Epoch seconds become timestamps, the recurring JSON blob becomes interval and usage-type columns, subscription items get joined to prices and products, and customer metadata becomes real columns you can group by. This is also where the daily snapshots live - the insert-only tables that solve limit number three.
Reports. Business-ready tables with the definitions your team agreed on: MRR by customer, MRR movements, a subscription KPI scorecard, invoice billings. One row per grain, documented, tested, and stable enough for a dashboard or an AI analyst to sit on top.
The layering is not bureaucracy. It is what lets you change a metric definition without re-ingesting anything, and re-ingest a source without touching a metric.
There are a lot of ways to build this, and most of them work.
| Approach | Good fit | Trade-off |
|---|---|---|
| Managed ELT (Fivetran, Airbyte, Stitch, Hevo) | You want the Stripe sync operated by someone else | Replication only - modelling, checks, and orchestration live elsewhere, and pricing tracks volume |
| Stripe Data Pipeline | You already use Snowflake, Redshift, Databricks, or BigQuery and want the lowest-effort load | Raw tables only, and it is a Stripe add-on rather than a general ingestion path |
| dbt + a connector + an orchestrator | You already run this stack and have the team for it | Three tools, three configs, three failure modes, and ingestion is somebody else's problem |
| Subscription analytics SaaS (ChartMogul, Baremetrics, ProfitWell) | You want SaaS metrics tomorrow with no engineering | Their definitions, their surface, and the data does not land somewhere you can join it |
| Custom Stripe API scripts | Genuinely unusual extraction logic | You own pagination, rate limits, retries, schema drift, and backfills forever |
| Bruin | You want ingestion, SQL models, quality checks, lineage, and scheduling as files in one repo | Code-first, so it suits teams comfortable with Git and SQL |
The comparison people ask for is Bruin against the middle two rows, and the honest version is this: if you already have Fivetran and dbt working and your team is happy, the marginal gain from switching is small. Bruin's argument is for the team that does not have that stack yet and is deciding what to assemble. In that situation, one CLI that does ingestion, transformation, checks, and lineage in a single project means one thing to learn, one thing to run in CI, and one place a coding agent has to look to understand your pipeline.
Where the difference is sharper: ingestr is open source, the Bruin CLI is open source, and running the whole thing locally costs nothing. You can find out whether the reports are worth having before you pay anyone.
The failure mode for a first Stripe pipeline is almost never "not sophisticated enough." It is "took four weeks and nobody uses it."
That happens because the first version tries to be the final version: every Stripe object, a semantic layer, three BI tools, a governance model, and a Slack alert. Meanwhile the CFO still cannot see net revenue retention by segment.
The version that works looks like this:
bruin run on your laptop against your own warehouse.The point of a template is that it collapses step one and two into a single command. You are not evaluating an architecture in the abstract; you are looking at real numbers from your own Stripe account within the hour, and then deciding what to change.
bruin init stripe-bigquery writes 19 assets across the three layers, plus a dashboard.
stripe_raw - six ingestr assets, one per Stripe resource. Incremental discovery on Stripe's created field, merge on write.
stripe_stage - nine models. Seven conformed entities plus two insert-only daily snapshots. The MRR rules are explicit and readable: an item counts only if the subscription is active or past due and the price is recurring, not metered, above zero, and billed monthly or annually. Annual prices are divided by twelve. Anything excluded carries a labelled reason - trialing_subscription, metered_price, zero_amount_price, unsupported_cadence - so you can query why a subscription contributes nothing instead of guessing.
stripe_reports - four tables:
| Report | Answers |
|---|---|
monthly_mrr_by_customer | Who pays what, with CRM segment, region, and sales owner joined in from Stripe customer metadata |
monthly_mrr_movements | New, reactivation, expansion, contraction, churn - classified after summing a customer's subscriptions, so a plan swap does not show up as churn plus new business |
monthly_subscription_kpis | Ending MRR, run-rate ARR, customer counts, net revenue retention by currency |
monthly_invoice_billings | Non-draft, non-void billings by finalization month, with a labelled creation-date fallback |
dashboards/stripe-billing-analytics.yml - a Dashboards as Code file you serve on localhost:8321. Metric tiles, an MRR line chart, a stacked movement chart where contraction and churn plot negative, and a top-accounts table.
Two things the template does that are worth stealing even if you build your own. First, the movement report has a custom check that enforces the roll-forward: beginning MRR plus new, reactivation, expansion, contraction, and churn must equal ending MRR. Break the logic and the run fails instead of publishing a wrong number. Second, every money column stays in native minor units and nothing is summed across currencies, because the template ships no FX policy and refuses to pretend it has one.
The step-by-step guide walks the whole thing in about 20 minutes, excluding ingestion time.
The template targets BigQuery, but only three things are actually BigQuery-specific: the asset type (bq.sql), the ingestr destination in pipeline.yml, and a handful of dialect functions - TIMESTAMP_SECONDS, JSON_VALUE, SAFE_DIVIDE, SAFE_CAST, COUNTIF, QUALIFY.
The layering, the model structure, the snapshot strategy, the MRR rules, the checks, and the metric policy are all portable SQL thinking. ingestr already supports Snowflake, Databricks, ClickHouse, Redshift, Postgres, DuckDB, and more as destinations, so the raw layer is a one-line change.
Which makes the port a good agent task. Hand your coding agent the repo and something like:
Refactor this pipeline from BigQuery to Snowflake. Keep the three-layer structure, asset names, column names, materialization strategies, dependencies, checks, and metric definitions exactly as they are - only the dialect and platform config change. Change
bq.sqltosf.sql, the ingestr destination tosnowflake, and translateTIMESTAMP_SECONDS,JSON_VALUE,SAFE_DIVIDE,SAFE_CAST,COUNTIF, andQUALIFYto their Snowflake equivalents. Work one directory at a time and runbruin validate --fastafter each.
That works because the agent has the whole thing in front of it as files. A pipeline defined in a vendor UI cannot be refactored by anyone, human or otherwise.
Local is not a permanent home. But it is the right first home, and moving off it later is a deployment change rather than a rewrite - same repo, same bruin run, credentials injected from wherever you keep secrets.
| Option | Good when | You operate |
|---|---|---|
| GitHub Actions | You already use GitHub and a daily cron is enough | Secrets, the workflow, and its logs |
| Self-hosted (cron, Airflow, ECS, Cloud Run) | You need a VPC, private connectivity, or an existing scheduler | The host, the scheduler, and alerting |
| Bruin Cloud | You want scheduling, lineage, run history, alerting, and SSO without building them | Nothing |
Pick based on what you want to own, not on what sounds most serious.
A pipeline that lies confidently is worse than a dashboard that admits its limits. Three disclaimers belong in your models, not just in someone's head:
MRR is not revenue. It is a gross list-price run rate. It is not recognized revenue, not bookings, not cash. If the template's version excludes discounts and metered usage and yours should not, change it - but write down which you chose.
Money does not add across currencies. Minor units in, minor units out, one currency at a time until you add an FX rate source and a stated conversion policy. A single number that quietly mixes USD and EUR is worse than two honest numbers.
The first month is a baseline. Movement and retention need two contiguous monthly observations. On day one they are empty. Say so on the dashboard rather than rendering a zero that looks like a fact.
A Stripe analytics pipeline loads Stripe billing objects into a data warehouse, transforms them into typed models with your own metric definitions, and publishes report tables such as MRR by customer, MRR movements, subscription KPIs, and invoice billings. It runs on a schedule and keeps a point-in-time history that Stripe itself does not retain.
Stripe only knows about Stripe, so it cannot join billing to product usage, CRM, support, or marketing data. Its metric definitions are fixed. Sigma is read-only SQL inside the Dashboard rather than a warehouse you can point BI tools at. And none of these surfaces keep the daily observation history that churn and retention require.
No. Stripe's public API exposes the same billing objects, and open-source tools such as ingestr can load them into BigQuery, Snowflake, ClickHouse, Databricks, Redshift, Postgres, or DuckDB. Sigma is genuinely useful for ad-hoc SQL inside Stripe, and Data Pipeline is a low-effort managed load. Neither is required to own your metric definitions.
Normalize every recurring price to a monthly amount, multiply by the subscription item quantity, and exclude one-time, metered, and zero-amount prices. Decide explicitly whether discounts, tax, and proration are in or out. Keep amounts in native minor units. Then snapshot daily, because Stripe subscriptions are mutable and last month's MRR cannot be reconstructed after the fact.
Yes. Only the asset type, the ingestr destination, and the SQL dialect are BigQuery-specific. Everything structural is portable, and translating the dialect is a mechanical refactor a coding agent can do in one pass.
The Bruin CLI, ingestr, and DAC are open source and run locally at no cost. The only spend is warehouse storage and query usage, which is small for a billing-sized dataset. Deployment costs appear later, once the reports are actually being used.
Start with the Stripe Analytics Pipeline guide if you want the commands, or the template README if you would rather read the assets first.

Learn how to load Shopify data into ClickHouse, choose an ingestion method, model a bronze-silver-gold ecommerce warehouse, and use Bruin to spend less time maintaining data tooling.

What is CDC in data engineering? Learn how change data capture records database changes, differs from ETL and incremental loading, and when to use it.

What is CDC streaming? Learn how it reads database logs, where Kafka fits, how it differs from batch CDC, and when its operational cost is worth it.
Practical updates on open-source data pipelines, AI analysts, governance, and what we are shipping at Bruin.
The signup form is hosted by Brevo. Accept cookies to load it.