TL;DR: Bruin is the best fit for Python teams that want the whole pipeline in one project: you write a Python asset whose materialize() function returns a DataFrame or yields rows, Bruin loads the result into your warehouse, and the SQL transformations and quality checks run right next to it. If you want a pure Python library to import into your own code, dlt is the strongest one, with schema inference, incremental loading, and state built in. PyAirbyte is best when you want to run Airbyte's connector catalog from Python. The Singer SDK is best for building reusable open connectors. For routine loads with no custom logic, ingestr, Bruin's ingestion CLI, moves a source into a warehouse with one command; it is written in Go, and its pip package lets you call it from Python. Reach for a library when you need custom logic in the pipeline; reach for a tool when you just need the data moved.
If you searched for a "Python data ingestion library," you probably want one of two different things, and they have different best answers:
- "I want to write pipeline logic in Python" (custom sources, transformations mid-flight, importable functions you can unit test). You want a real library: dlt. If the transformations and checks should live in the same project, Bruin Python assets do the same job inside a pipeline.
- "I use Python and I just need this source in my warehouse" (no appetite to write and maintain a pipeline). You want a tool you can install and run, not a library: ingestr, or PyAirbyte if you need Airbyte's catalog.
Most "best Python library" questions are secretly the second one. Let us cover both honestly.
We build Bruin and ingestr, so you know where we stand. We also say plainly where another tool is the better pick: if you want a pure Python library inside your own application, dlt usually is.
Key takeaways
- Decide first: a library you import, or a tool you run. Custom extraction logic wants a library like dlt; a routine "copy this table" job wants a command.
- Of the 5 options here, dlt is the strongest pure-Python ingestion library. It is Apache 2.0, runs anywhere Python runs, and handles schema inference, incremental loading, and pipeline state for you.
- Since version 1, ingestr is a Go binary, not a Python library.
pip install ingestrdownloads the binary, and thesdkextra streams Python rows and DataFrames to it as Arrow. - Loading is 1 of 4 jobs. You still transform, test, and schedule the data; Bruin covers all 4 from one project, while a loading library leaves 3 of them to other tools.
How we compared
- What you write: pipeline code, connector code, an asset file, or a single command.
- What runs underneath: pure Python in your process, a connector runtime, or a compiled binary.
- How incremental loads work: where state lives and what happens on a rerun.
- What happens after the load: transformations, quality checks, and scheduling, or nothing.
- Licence, read from each project's repository or package metadata.
Facts about each tool come from its public documentation, checked in November 2026. Facts about Bruin come from our docs and pricing page.
The 5 options at a glance
| Option | Form factor | Licence | You write | Best for |
|---|---|---|---|---|
| Bruin (Python assets + ingestr) | Pipeline CLI that runs Python and SQL assets; ingestr is a Go CLI with a Python SDK | Bruin CLI Apache 2.0; ingestr FSL (source-available) | A materialize() function, an asset file, or one command | Moving a source into a warehouse with one command, and running the SQL and Python transformations on it in the same project |
| dlt | Python library (pip install dlt) | Apache 2.0 | Pipeline code | Custom, testable pipelines in Python |
| PyAirbyte | Python library (pip install airbyte) | ELv2 | Glue code around Airbyte connectors | Running Airbyte's catalog from Python |
| Singer SDK | Python framework (Meltano Singer SDK) | Apache 2.0 | Taps and targets | Building reusable open connectors |
| pandas + SQLAlchemy | General libraries | BSD-3-Clause and MIT | Everything yourself | Tiny one-off loads |
Best Python ingestion option by need:
- Ingestion, SQL and Python transformations, and quality checks in one project: Bruin
- A routine load with no custom logic, from the shell or from Python: ingestr
- Custom extraction logic inside your own Python code: dlt
- A connector from Airbyte's catalog, driven from Python: PyAirbyte
- A reusable Singer connector for others: Singer SDK
- A tiny, throwaway load: pandas + SQLAlchemy
The 5 options, one by one
1. Bruin (Python assets and ingestr)
What it is: Bruin CLI is an open-source (Apache 2.0) pipeline tool that runs SQL, Python, R, and ingestr assets from one project. A Python asset defines a materialize() function that returns a pandas or Polars DataFrame, a PyArrow table, or a list of dicts, or yields rows or Arrow tables, and Bruin loads the result into the warehouse through ingestr. For routine loads, ingestr also runs on its own: one command from a source URI to a destination URI, with incremental loading built in.
ingestr ingest \
--source-uri 'postgresql://user:pass@host/db' \
--source-table 'public.orders' \
--dest-uri 'bigquery://project-id' \
--dest-table 'raw.orders'
Where it wins: ingestion, transformations, quality checks, and lineage live in one Git project instead of a dlt script here, a dbt project there, and Airflow gluing it together. Bruin installs and isolates Python versions and dependencies per asset with uv, so two assets can run different Python versions, and quality checks run on a Python asset just as they do on a SQL asset. ingestr handles incremental loading and schema handling, which is exactly the code you would otherwise write (and maintain) with a library. See Bruin ingestion.
Where it falls short: Bruin is not a library you import into an existing application; you adopt its project layout. Python materialization is newer than dlt's, and Bruin's docs still flag it as recently introduced. ingestr is source-available under FSL rather than an OSI open-source licence, and its Python SDK does not support async generators yet.
Best for: Python teams that want ingestion, transformations, and quality checks in one project, and teams that just need a source moved with one command.
2. dlt
What it is: an open-source (Apache 2.0) Python library that loads data from messy sources into structured datasets. You pip install dlt, define a source as decorated Python functions, and it handles schema inference, incremental loading, state, and loading into your destination.
import dlt
@dlt.resource(write_disposition="merge", primary_key="id")
def orders(updated_after=dlt.sources.incremental("updated_at")):
yield from fetch_orders(since=updated_after.last_value)
pipeline = dlt.pipeline(destination="bigquery", dataset_name="raw")
pipeline.run(orders())
Where it wins: dlt is the strongest true Python ingestion library. It is code-first, testable, runs anywhere Python runs (Airflow, serverless functions, or any other cloud deployment), and gives you full control when a source is weird or you need to reshape data as it flows. Its REST API source covers many APIs without writing a client from scratch. If you need custom logic inside the ingestion path, dlt is the pick.
Where it falls short: you own pipeline code: more power, more to maintain. Incremental loading depends on state that dlt stores in the destination (a _dlt_pipeline_state table), which is one more thing to reason about when runs move between environments. Transformations, checks, and scheduling come from other tools or from dltHub's commercial platform.
Best for: custom, testable pipelines in pure Python.
3. PyAirbyte
What it is: Airbyte's Python library (pip install airbyte, licensed ELv2) for running Airbyte connectors in a Python process. Declarative YAML connectors run directly, Python connectors install automatically, and records land in a cache, a local DuckDB file by default, with Postgres, BigQuery, Snowflake, and MotherDuck as other options.
Where it wins: it is the right choice when the connector you need already exists in Airbyte's catalog and you want to drive it from Python. Airbyte recommends it for notebooks and early prototypes where you do not need to run a server, so you get catalog breadth without standing up the platform.
Where it falls short: Java-based destinations need Docker, data lands in a cache before you move it on, and ELv2 is a source-available licence rather than an OSI open-source one. PyAirbyte is not an SDK for managing Airbyte; for that, Airbyte points you to its API and Python SDK.
Best for: running Airbyte's catalog from Python.
4. Singer SDK
What it is: the Meltano Singer SDK (Apache 2.0), a Python framework for building taps (sources) and targets (destinations) that follow the Singer spec.
Where it wins: reach for it when you are authoring a connector meant to be reused, not when you just want to move data today. It gives you base classes for REST, GraphQL, and SQL streams, authenticators, and targets, so a new connector follows the spec without starting from zero.
Where it falls short: it is infrastructure for the ecosystem, not a quick ingestion path, and you still need a runner such as Meltano to configure and schedule taps and targets.
Best for: building reusable open connectors.
5. pandas + SQLAlchemy
What it is: two general-purpose libraries. For a genuinely tiny, one-off load, pd.read_sql(...) then df.to_sql(...) is enough.
Where it wins: it is fine for a throwaway job and needs no new dependency if pandas and SQLAlchemy are already in your environment.
Where it falls short: do not build a production pipeline this way: no incremental state, no schema evolution, no restartability, and to_sql is slow at volume. It is a script, not a pipeline.
Best for: tiny one-off loads.
Is ingestr a Python library?
No. ingestr started in Python, and for v1 Bruin rewrote it in Go. Today it is a single compiled binary, installed with a script or with pip, and source-available under the Functional Source License (FSL-1.1-ALv2), which is free for internal use and converts each release to Apache 2.0 two years after it ships.
The pip package is a thin Python wrapper around that binary. On first use it downloads and caches the matching release binary, verified against checksums in the package. That gives Python users three ways in:
- Run the CLI.
pip install ingestr, theningestr ingest ...from a shell, a subprocess, or an AirflowBashOperator. - Call it from Python.
ingestr.run([...])passes CLI arguments through, andingestr.run_cli(...)maps keyword arguments toingestr ingestflags. - Push Python data into it. With
pip install 'ingestr[sdk]',ingestr.ingest()accepts rows, generators, pandas or Polars DataFrames, and PyArrow tables, and streams them to the binary as Arrow.
import ingestr
def events():
yield [{"id": 1, "event": "signup"}, {"id": 2, "event": "purchase"}]
ingestr.ingest(
events,
dest_uri="postgresql://user:pass@localhost:5432/app",
dest_table="public.events",
)
So you can drive ingestr from Python, but the ingestion work runs in Go, outside your Python process. On Linux the release binaries need glibc, so Alpine-based images must provide their own binary. Inside a Bruin pipeline you do not call it at all: a Python asset's materialize() output is loaded through ingestr for you.
How do I build a data pipeline in Python?
A data pipeline in Python has four steps: extract and load the data, transform it, check it, and schedule it. A library like dlt covers the first step well. The rest comes from other tools, unless you use a pipeline tool that covers all four.
In Bruin, the extract-and-load step is a Python asset. It lives in a project next to SQL models that read from it:
"""@bruin
name: raw.orders
image: python:3.13
connection: bigquery
materialization:
type: table
strategy: merge
columns:
- name: id
type: integer
primary_key: true
checks:
- name: not_null
@bruin"""
import requests
def materialize():
page = 1
while True:
rows = requests.get(f"https://api.example.com/orders?page={page}").json()["items"]
if not rows:
break
yield rows
page += 1
Each yielded page is written to disk as its own Arrow batch, so a large API pull never has to fit in memory. bruin run loads the data, runs the downstream SQL, and runs the checks; Bruin Cloud schedules the pipeline if you do not want to run your own scheduler. For a step-by-step walkthrough, see how to build your first ELT pipeline with SQL and Python.
A library is one step; the pipeline is the job
Even the best Python ingestion library only handles extract-and-load. You still need to model the raw data, check its quality, and schedule it. ingestr is the ingestion layer of Bruin, whose open-source CLI runs the Python (and SQL) transformations and quality checks next to ingestion, so a Python team does not end up with a dlt script here, a dbt project there, and Airflow gluing it together. If you prefer a pure library approach, that is valid too, just be clear-eyed that the library is one of four things you will end up running.
Which Python ingestion option should you pick?
| Your situation | Pick | Why |
|---|---|---|
| You want ingestion, transformations, and quality checks in one Python and SQL project | Bruin | Python assets are loaded for you, with checks and lineage beside them |
| You just need a source in the warehouse with minimal code | ingestr | One command, or one function call from Python through its SDK |
| You need custom logic inside the ingestion path | dlt | A pure Python library with schema inference, incremental loading, and state |
| You need a specific Airbyte connector, from Python | PyAirbyte | Airbyte's catalog without running a server |
| You are building a connector for others to reuse | Singer SDK | The standard way to write Singer taps and targets |
| The load is one-off, tiny, and throwaway | pandas + SQLAlchemy | No new tools, as long as it stays a script |
When Bruin is not the right pick
If you are embedding ingestion inside your own Python application or service, import dlt: Bruin is a project you run, not a library you call. If the connector you need only exists in Airbyte's catalog, PyAirbyte is the shorter path. If your policy requires an OSI-approved licence for the ingestion engine, dlt (Apache 2.0) fits, while ingestr is source-available. Bruin earns its place when the load is one step in a pipeline you also want to transform, test, and schedule from the same project. For a side-by-side with dlt, see dlt alternatives.
Frequently asked questions
What is the best Python library for data ingestion?
Bruin is the best fit when you want the whole pipeline, ingestion plus transformations and quality checks, in one Python and SQL project. If you want a pure Python library to import into your own code, dlt is the strongest option.
Is ingestr a Python library?
No, ingestr is a Go command-line tool, source-available under FSL, and its pip package downloads the binary and adds a Python SDK, so you can call it from Python while the ingestion runs in Go. Bruin uses the same engine to load the output of Python assets.
How do I build a data pipeline in Python?
Extract and load with a library like dlt or a Bruin Python asset, then transform, test, and schedule the result. Bruin keeps all four steps in one project, with Bruin Cloud running the schedule, so you do not glue dlt, dbt, and Airflow together.
Can I use ingestr from Python code?
Yes. Install ingestr[sdk] and call ingestr.ingest() with rows, a generator, a DataFrame, or a PyArrow table, or use ingestr.run() to run any CLI command; inside Bruin, a Python asset's output goes through ingestr automatically.
What is the fastest data ingestion tool for Python users?
It depends on the source, destination, and data size, so test on your own data. Bruin publishes reproducible benchmarks comparing ingestr with Sling, dlt, and Airbyte on its ingestr benchmarks page.
Is dlt free?
Yes, dlt is open source under Apache 2.0, and dltHub sells a managed platform on top of it. Bruin's CLI is also Apache 2.0, and Bruin Cloud starts free on the Team plan.
Should I use pandas to load data into a warehouse?
Only for tiny one-off loads, because to_sql has no incremental state or schema evolution and is slow at volume. For anything recurring, use dlt, ingestr, or a Bruin Python asset, which loads a returned DataFrame for you.
What is the difference between dlt and Bruin?
dlt is a Python library for loading data that you deploy and schedule yourself. Bruin is a pipeline tool where a Python asset's output is loaded for you and sits next to SQL models, quality checks, and lineage.
For the wider picture see the best data ingestion tools in 2026, the fastest data ingestion tools compared, and Python vs SQL: choosing the right tool. To see what Bruin Cloud adds, check pricing.
I work at Bruin and we make ingestr, so this is an informed but interested take. Corrections welcome at [email protected].