Build an ETL Pipeline with Airbyte Cloud and dbt Core for SaaS Analytics

Stand up a complete extract, load, and transform (ELT) pipeline that ingests SaaS data with Airbyte Cloud and models it with dbt, all coordinated by Kestra. This blueprint solves the classic analytics engineering gap: keeping managed ingestion and version-controlled transformations in lockstep so that dbt never runs against stale or half-loaded tables. It fans out three Airbyte Cloud connection syncs in parallel (Salesforce, Google Analytics, and Facebook Ads), waits for every load to finish, then clones a dbt project and runs a dbt build against BigQuery to produce analytics-ready models.

How it works

1. The data_ingestion task is an io.kestra.plugin.core.flow.Parallel block that triggers three io.kestra.plugin.airbyte.cloud.jobs.Sync tasks at once: salesforce, google_analytics, and facebook_ads, each pointed at its own Airbyte Cloud connectionId.
2. The Airbyte API token is injected once for all three syncs through pluginDefaults on io.kestra.plugin.airbyte.cloud.jobs.Sync, so credentials stay out of the task bodies.
3. The Parallel block only completes once all three syncs return, which gates transformation until the raw data has landed.
4. The dbt task is an io.kestra.plugin.core.flow.WorkingDirectory. Inside it, io.kestra.plugin.git.Clone checks out the dbt project from a Git repository on the main branch.
5. io.kestra.plugin.dbt.cli.Build then runs the dbt build inside a Docker taskRunner using the ghcr.io/kestra-io/dbt-bigquery image. A profiles.yml and a sa.json keyfile are supplied as inputFiles, with the BigQuery service account passed in from a secret.

What you get

• Parallel ingestion across multiple SaaS sources in a single run.
• A hard ordering guarantee: dbt only builds after every Airbyte sync succeeds.
• Version-controlled transformations cloned fresh from Git on each run.
• A reproducible, containerized dbt runtime, no local dbt install required.
• Analytics-ready tables materialized in BigQuery.

Who it's for

• Analytics engineers who own dbt models and want them refreshed after ingestion.
• Data platform teams standardizing ELT across many SaaS sources.
• Teams combining managed Airbyte Cloud ingestion with self-managed dbt.

Why orchestrate this with Kestra

Airbyte Cloud schedules syncs and dbt Cloud schedules builds, but neither can natively guarantee that dbt starts only after a specific set of loads has finished. Kestra closes that gap with a declarative YAML pipeline: the Parallel block plus WorkingDirectory express the dependency explicitly, retries can be added per task, event or schedule triggers can drive the run, and every execution is captured for lineage and replay. You get one source of truth for the whole ELT flow instead of two disconnected schedulers.

Prerequisites

• An Airbyte Cloud workspace with the three connections configured.
• A BigQuery project and dataset, plus a service account with write access.
• A dbt project in a Git repository (this example uses a public demo repo).

Secrets

• AIRBYTE_CLOUD_API_TOKEN: API token used to authenticate the Airbyte Cloud syncs.
• GCP_CREDS: GCP service account JSON written to sa.json for the dbt BigQuery profile.

Quick start

1. Add the AIRBYTE_CLOUD_API_TOKEN and GCP_CREDS secrets to your Kestra instance.
2. Replace the three connectionId values with your own Airbyte Cloud connections.
3. Point io.kestra.plugin.git.Clone at your dbt repository and branch.
4. Update the profiles.yml dataset, project, and location to match your BigQuery setup.
5. Run the flow and confirm the dbt models appear in your warehouse.

How to extend

• Add more Airbyte connections inside the Parallel block to ingest additional sources.
• Add a Schedule trigger to refresh data on a cadence, or a webhook trigger for event-driven runs.
• Swap the BigQuery profile and Docker image for Snowflake, Postgres, or another warehouse.
• Add downstream tasks (alerts, dashboard refreshes, data quality checks) after the dbt build.

Links

• Airbyte plugin docs: https://kestra.io/plugins/plugin-airbyte
• dbt plugin docs: https://kestra.io/plugins/plugin-dbt
• Git plugin docs: https://kestra.io/plugins/plugin-git
• Airbyte orchestration guide: https://kestra.io/orchestration/airbyte