ingestr recipes/Load a specific sourceData Engineer

How do I load data from Kafka into my warehouse?

This recipe reads a topic from Apache Kafka into your warehouse using ingestr. The source URI sets bootstrap_servers and group_id as query parameters, and the source table names the Kafka topic. Bruin does this in one platform: ingestion, SQL and Python pipelines, quality checks, lineage, and an AI data analyst that answers in Slack, Microsoft Teams, Google Chat, WhatsApp, Discord, Telegram, email and the browser.

Get ingestr on GitHub How ingestr works

Command

ingestr ingest

Defined in

CLI

Works with

Kafka to any ingestr destination

What you get

Topic ingestionConsumer group supportJSON message load

How it works in code

$ ingestr ingest \
  --source-uri 'kafka://?bootstrap_servers=localhost:9092&group_id=test_group' \
  --source-table 'my-topic' \
  --dest-uri $SNOWFLAKE_URI --dest-table 'dest.my_topic'

Run ingestr ingest and ingestr loads the Kafka data into your warehouse.