Build an ETL Pipeline with Parallel Airbyte Syncs and dbt Core Transformations

Combine self-hosted Airbyte ingestion with dbt Core transformations into a single, dependency-aware ELT pipeline. This blueprint ingests data from multiple SaaS sources in parallel, waits for every sync to finish, then runs a dbt build to turn raw tables into analytics-ready models in your cloud data warehouse. It solves the classic ingestion-then-transform race condition: dbt only runs once all Airbyte connections have landed fresh data, so your BI layer never reads half-loaded tables.

How it works

1. A data_ingestion task of type io.kestra.plugin.core.flow.Parallel launches three io.kestra.plugin.airbyte.connections.Sync tasks at once: salesforce, google_analytics, and facebook_ads, each targeting its own Airbyte connectionId.
2. The Parallel task acts as a barrier, so the flow only proceeds once all three syncs complete successfully.
3. A dbt task of type io.kestra.plugin.core.flow.WorkingDirectory sets up an isolated workspace, then io.kestra.plugin.git.Clone pulls the dbt project from a Git repository.
4. io.kestra.plugin.dbt.cli.Build runs dbt build inside a Docker taskRunner using the ghcr.io/kestra-io/dbt-bigquery image, with the BigQuery profiles.yml and service-account sa.json injected as inputFiles.

What you get

• Parallel ingestion that cuts total sync wall-clock time versus running connections one by one.
• A hard ordering guarantee between ingestion and transformation.
• Reproducible dbt runs in a pinned container image, isolated per execution.
• Credentials kept out of YAML via Kestra secrets.

Who it's for

• Analytics engineers building dbt models on freshly ingested data.
• Data platform teams running self-hosted Airbyte who want full control over ingestion and transformation.
• BI teams who need consistent, dependency-correct refreshes feeding dashboards.

Why orchestrate this with Kestra

Airbyte schedules connections independently and dbt Cloud schedules jobs independently, but neither natively coordinates ingestion finishing before transformation starts across tools. Kestra closes that gap with the Parallel barrier and declarative WorkingDirectory sequencing. You also get event and schedule triggers, automatic retries on transient sync or warehouse failures, full execution lineage across both tools, and the entire pipeline defined as version-controlled YAML.

Prerequisites

• A reachable self-hosted Airbyte server and valid connectionId values for each source.
• A dbt project in a Git repository (this example uses a public one) and a target cloud warehouse.
• A Docker task runner available to the Kestra worker.

Secrets

• AIRBYTE_USERNAME: Airbyte API/basic-auth username.
• AIRBYTE_PASSWORD: Airbyte API/basic-auth password.
• GCP_CREDS: GCP service-account JSON, injected as sa.json for the dbt BigQuery profile.

Quick start

1. Add the AIRBYTE_USERNAME, AIRBYTE_PASSWORD, and GCP_CREDS secrets to your Kestra instance.
2. Replace the three placeholder connectionId values with your real Airbyte connection IDs.
3. Update the Airbyte url in pluginDefaults and point clone_repository at your dbt project.
4. Edit the profiles.yml dataset, project, and location to match your warehouse.
5. Execute the flow and confirm both ingestion and the dbt build succeed.

How to extend

• Add or remove Sync tasks inside Parallel to match your source list.
• Swap the container image and profiles.yml to target Snowflake, Postgres, or another warehouse.
• Add a Schedule or flow trigger to refresh on a cadence or react to upstream events.
• Append data-quality or notification tasks after dbt_build to alert on failures.

Links

• plugin-airbyte docs
• Airbyte orchestration with Kestra