Source
yaml
id: airbyte-cloud-dbt
namespace: company.team
tasks:
- id: data_ingestion
type: io.kestra.plugin.core.flow.Parallel
tasks:
- id: salesforce
type: io.kestra.plugin.airbyte.cloud.jobs.Sync
connectionId: e3b1ce92-547c-436f-b1e8-23b6936c12ab
- id: google_analytics
type: io.kestra.plugin.airbyte.cloud.jobs.Sync
connectionId: e3b1ce92-547c-436f-b1e8-23b6936c12cd
- id: facebook_ads
type: io.kestra.plugin.airbyte.cloud.jobs.Sync
connectionId: e3b1ce92-547c-436f-b1e8-23b6936c12ef
- id: dbt
type: io.kestra.plugin.core.flow.WorkingDirectory
tasks:
- id: clone_repository
type: io.kestra.plugin.git.Clone
url: https://github.com/kestra-io/dbt-demo
branch: main
- id: dbt_build
type: io.kestra.plugin.dbt.cli.Build
runner: DOCKER
dbtPath: /usr/local/bin/dbt
dockerOptions:
image: ghcr.io/kestra-io/dbt-bigquery:latest
inputFiles:
.profile/profiles.yml: |
jaffle_shop:
outputs:
dev:
type: bigquery
dataset: your_big_query_dataset_name
project: your_big_query_project
fixed_retries: 1
keyfile: sa.json
location: EU
method: service-account
priority: interactive
threads: 8
timeout_seconds: 300
target: dev
sa.json: "{{ secret('GCP_CREDS') }}"
pluginDefaults:
- type: io.kestra.plugin.airbyte.cloud.jobs.Sync
values:
token: "{{ secret('AIRBYTE_CLOUD_API_TOKEN') }}"
About this blueprint
Parallel Ingest BigQuery Git dbt
This flow runs Airbyte Cloud syncs in parallel and then runs dbt Core's CLI commands.
It's recommended to configure the GCP service account and Airbyte API token, referenced in the pluginDefaults
, as a secret.
More Related Blueprints