Bruin - Notion to Bigquery Template

This pipeline is a simple example of a Bruin pipeline that copies data from Notion to BigQuery. It demonstrates how to use the bruin CLI to build and run a pipeline.

The pipeline includes two sample assets already:

• raw.notion: A simple ingestr asset that takes copies a table from Notion to BigQuery
• myschema.example: A simple SQL asset that creates a table in BigQuery.
  • Feel free to change the type from bq.sql to anything.

Setup

The pipeline already includes an empty .bruin.yml file, fill it with your connections and environments. You can read more about connections here.

Here's a sample .bruin.yml file:

environments:
  default:
    connections:
      google_cloud_platform:
        - name: "gcp"
          service_account_file: "/path/to/my/key.json"
          project_id: "my-project-dev"
      notion:
        - name: "my-notion-connection"
          api_key: "XXXXXXXX"

Running the pipeline

Bruin CLI can run the whole pipeline or any task with the downstreams:

bruin run assets/notion.asset.yml

Starting the pipeline execution...

[2023-03-16T18:25:14Z] [worker-0] Running: dashboard.bruin-test
[2023-03-16T18:25:16Z] [worker-0] Completed: dashboard.bruin-test (1.681s)
[2023-03-16T18:25:16Z] [worker-4] Running: hello
[2023-03-16T18:25:16Z] [worker-4] [hello] >> Hello, world!
[2023-03-16T18:25:16Z] [worker-4] Completed: hello (116ms)

Executed 2 tasks in 1.798s

You can also run a single task:

bruin run assets/hello.py

Starting the pipeline execution...

[2023-03-16T18:25:59Z] [worker-0] Running: hello
[2023-03-16T18:26:00Z] [worker-0] [hello] >> Hello, world!
[2023-03-16T18:26:00Z] [worker-0] Completed: hello (103ms)


Executed 1 tasks in 103ms

You can optionally pass a --downstream flag to run the task with all of its downstreams.

That's it, good luck!