Trigger multiple Fivetran syncs in parallel, then run a dbt job

Orchestrate a full ELT pipeline that loads data from multiple sources with Fivetran and transforms it with dbt Core. This blueprint fans out four Fivetran connector syncs (Salesforce, Stripe, Google Analytics, and Facebook Ads) so they run concurrently, then runs a dbt build against the freshly loaded BigQuery warehouse. It solves the common ELT timing problem: dbt should only transform data once every upstream extract has finished loading, and running syncs one after another wastes time when they are independent.

How it works

1. The data_ingestion task uses io.kestra.plugin.core.flow.Parallel to launch four io.kestra.plugin.fivetran.connectors.Sync tasks at once: salesforce, stripe, google_analytics, and facebook_ads, each targeting its own connectorId. Shared apiKey and apiSecret come from pluginDefaults.
2. Once every sync completes, the dbt_core task runs inside io.kestra.plugin.core.flow.WorkingDirectory so cloned files and built artifacts share one workspace.
3. clone_repository uses io.kestra.plugin.git.Clone to pull the dbt project from the dbt-demo repository on the main branch.
4. dbt_build runs io.kestra.plugin.dbt.cli.Build on a Docker taskRunner using the ghcr.io/kestra-io/dbt-bigquery image. It injects a profiles.yml pointing at BigQuery and an sa.json service-account key sourced from a secret.

What you get

• Parallel extraction that finishes in the time of the slowest connector, not the sum of all four.
• A clean dependency boundary so dbt transforms only run after every load succeeds.
• Reproducible dbt runs in a pinned, containerized environment.
• Secrets kept out of the flow definition.

Who it's for

• Analytics engineers building ELT pipelines on BigQuery.
• Data teams already using Fivetran connectors and dbt Core together.
• Platform engineers consolidating extract and transform steps under one orchestrator.

Why orchestrate this with Kestra

Fivetran and dbt Cloud each schedule their own jobs, but neither can express a cross-tool dependency where dbt waits on four independent Fivetran syncs and only then transforms. Kestra closes that gap: declarative YAML defines the Parallel fan-out and the sequential dbt step in one place, event triggers can start the pipeline on a schedule or webhook, retries and error handling protect each task, and full run lineage shows exactly which sync or model failed.

Prerequisites

• A Fivetran account with the four connectors configured and their connector IDs.
• A BigQuery project and dataset, plus a GCP service-account key with access.
• A dbt Core project (this blueprint clones kestra-io/dbt-demo).

Secrets

• FIVETRAN_API_KEY: Fivetran API key used by every Sync task.
• FIVETRAN_API_SECRET: Fivetran API secret used by every Sync task.
• GCP_CREDS: GCP service-account JSON injected as sa.json for the dbt BigQuery profile.

Quick start

1. Add the three secrets above to your Kestra instance.
2. Replace each connectorId with your real Fivetran connector IDs.
3. Point clone_repository at your own dbt project and update profiles.yml with your BigQuery dataset and project.
4. Run the flow and watch the four syncs fan out before dbt build starts.

How to extend

• Add or remove Sync tasks inside the Parallel block as connectors change.
• Swap dbt build for dbt run plus dbt test, or split them into separate tasks.
• Add a schedule or flow trigger to run the pipeline automatically.
• Point the dbt profile at Snowflake, Postgres, or another warehouse by changing the image and profiles.yml.

Links

• Fivetran plugin documentation
• dbt plugin documentation
• dbt Core orchestration