How do I load data from Kafka into my warehouse?
This recipe reads a topic from Apache Kafka into your warehouse using ingestr. The source URI sets bootstrap_servers and group_id as query parameters, and the source table names the Kafka topic.
Command
ingestr ingestDefined in
CLI
Works with
Kafka to any ingestr destination
What you get
Topic ingestionConsumer group supportJSON message load
How it works in code
$ ingestr ingest \
--source-uri 'kafka://?bootstrap_servers=localhost:9092&group_id=test_group' \
--source-table 'my-topic' \
--dest-uri $SNOWFLAKE_URI --dest-table 'dest.my_topic'Run ingestr ingest and ingestr loads the Kafka data into your warehouse.
Related use cases
Load a specific source
Replicate a Postgres database into Snowflake
How do I replicate a Postgres database into Snowflake?
Load a specific sourceReplicate a database incrementally
How do I keep a database table in sync without full reloads (CDC-style)?
Load a specific sourceLoad Google Sheets into your warehouse
How do I load a Google Sheet into my warehouse?
Move data with one command
Open source. Source URI in, destination URI out, incremental by default.