All integrations
HTTP
+
Bruin

HTTP + Bruin

Source

Ingest HTTP data into your warehouse with incremental loading, quality checks, and full lineage. Defined in YAML, version-controlled in Git.

For business teams

What you get

  • API data, on schedule

    HTTP data lands in your warehouse automatically. No scripts to maintain, no pagination to handle.

  • Only fetch what changed

    Incremental sync means no re-processing. Bruin tracks watermarks so you only get new and updated records.

  • Catch API changes early

    Quality checks validate response data on every sync. Schema changes or missing fields get caught before they break models.

  • Transform in the same pipeline

    Reshape HTTP API data with SQL or Python. Compute metrics, normalize schemas, and build models, all version-controlled.

For data & engineering teams

How it works

  • Managed pagination & retries

    Bruin handles HTTP API pagination, rate limiting, and retries. You define the source, Bruin does the rest.

  • YAML-defined, Git-versioned

    Your HTTP pipeline is a YAML file. Review in PRs, deploy with CI/CD, roll back with git revert.

  • Incremental with watermarks

    Bruin tracks cursor positions and watermarks. Only new and updated HTTP records get fetched on each run.

  • Schema validation on responses

    Quality checks validate HTTP API response structure on every sync. Catch breaking API changes early.

Before you start

API endpoint URL
Authentication credentials (if required)

Step 1

Add your HTTP connection

Connect to any HTTP/REST API endpoint. Add this to your Bruin environment file, credentials are stored securely and referenced by name in your pipeline YAML.

Parameters

  • urlThe HTTP(S) endpoint URL
  • headersCustom headers as JSON (optional)
  • auth_typeAuthentication type (bearer, basic, etc.)
connections:
  http:
    type: http
    uri: "http://?url=<api_url>&headers=<headers>&auth_type=<auth_type>"

Step 2

Create your pipeline

Define a YAML asset that tells Bruin what to pull from HTTP and where to land it. This file lives in your Git repo, reviewable, version-controlled, and deployable with CI/CD. 

name: raw.http_data
type: ingestr

parameters:
  source_connection: http
  source_table: 'data'
  destination: bigquery

Step 3

Add quality checks

Add column-level and custom SQL checks to your HTTP data. If a check fails, the pipeline stops, bad data never reaches downstream models or dashboards.

Validate API data freshness on every sync
Ensure record IDs are unique across fetches
Catch missing fields from API response changes
columns:
  - name: id
    checks:
      - name: not_null
      - name: unique
  - name: fetched_at
    checks:
      - name: not_null

custom_checks:
  - name: API data is fresh
    query: |
      SELECT MAX(fetched_at) >
        TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 24 HOUR)
      FROM raw.http_data

Step 4

Run it

One command. Bruin connects to HTTP, pulls data incrementally, runs your quality checks, and lands clean data in your warehouse. If a check fails, the pipeline stops, bad data never reaches downstream.

Backfill historical data with --start-date
Schedule with cron or trigger from CI/CD
Full lineage from HTTP to your dashboards 
$ bruin run .
Running pipeline...

  http_data
    ✓ Fetched 2,847 new records
    ✓ Quality: campaign_id not_null     PASSED
    ✓ Quality: spend not_null           PASSED
    ✓ Quality: no negative ad spend     PASSED
    ✓ Loaded into bigquery

  Completed in 12s

Other API integrations

Allium

Allium

Anthropic

Anthropic

Frankfurter

Frankfurter

ISOC Pulse

ISOC Pulse

Socrata

Socrata

Ready to connect HTTP?

Start for free, or book a demo to see how Bruin handles ingestion, quality, lineage, and scheduling for your entire data stack.

Book a demo Read the docs