Can a pipeline step call an API and store the response?
Use a Python asset to fetch and shape the data, declare its dependencies, and Bruin materializes the result like any other model.
Command
bruin runDefined in
Python + YAML
Works with
Bruin CLI + your warehouse
What you get
First-class Python assetsisolated envswarehouse materialization
How it works in code
# scores.py
""" @bruin
name: ml.scores
materialization: {type: table}
@bruin """
import pandas as pd
def materialize():
return model.predict(load_features())Run bruin run and Bruin runs the Python asset in its own environment and lands the output in the warehouse.
Related use cases
Python in pipelines
Run a Python transformation as a first-class asset
Can Python models be first-class citizens next to SQL?
Python in pipelinesScore an ML model inside a pipeline
How do I run model scoring as part of a data pipeline?
Python in pipelinesShare a Python environment across assets
How do I manage Python dependencies for pipeline steps?
SQL and Python, one pipeline
Open source. Define assets, declare dependencies, run the graph with bruin run.