Closer-Every-Year

A batch pipeline tracking **gender gap indicators and relationship trends** (marriage, divorce, age at first marriage, pay gap) across European countries from 2005 to 2024, all sourced from the Eurostat API. **Bruin features I used:** • `type: python` assets for ingestion (Eurostat API → Parquet on GCS) • `type: bq.sql` and `type: duckdb.sql` assets for staging + analytics SQL transformations • `strategy: merge` on `(country, year)` — fully idempotent, no duplicate runs • Dependency resolution via asset references — no manual DAG wiring • Dual-environment setup: `local-pipeline` runs on DuckDB, `gcp-pipeline` runs on BigQuery — **same asset code, different connections** • `bruin run --environment cloud` for the GCP pipeline, `--workers 1` for local (DuckDB doesn't support concurrent writes) • Docker-based setup with the Bruin container + Terraform container side by side