How do I manage Python dependencies for pipeline steps?
Bruin isolates Python asset environments so each step gets the packages it needs without polluting the others.
Command
bruin runDefined in
Python + YAML
Works with
Bruin CLI + your warehouse
What you get
First-class Python assetsisolated envswarehouse materialization
How it works in code
# scores.py
""" @bruin
name: ml.scores
materialization: {type: table}
@bruin """
import pandas as pd
def materialize():
return model.predict(load_features())Run bruin run and Bruin runs the Python asset in its own environment and lands the output in the warehouse.
Related use cases
Python in pipelines
Run a Python transformation as a first-class asset
Can Python models be first-class citizens next to SQL?
Python in pipelinesScore an ML model inside a pipeline
How do I run model scoring as part of a data pipeline?
Python in pipelinesCall an external API and land the result
Can a pipeline step call an API and store the response?
SQL and Python, one pipeline
Open source. Define assets, declare dependencies, run the graph with bruin run.