Run dbt transformations on Google BigQuery from Git for analytics ETL

Run dbt Core transformations on Google BigQuery straight from a Git repository, turning version-controlled SQL models into analytics-ready tables and views. This blueprint solves a common analytics engineering problem: keeping dbt projects Git-managed and reproducible while running them on a real warehouse with proper authentication, logging, and dependency resolution, instead of relying on a developer laptop or an ad hoc cron job.

How it works

The flow uses a single io.kestra.plugin.core.flow.WorkingDirectory task named git to keep cloning and execution in one shared directory:

1. clone_repository (io.kestra.plugin.git.Clone) clones the kestra-io/dbt-example repository from the main branch into the working directory.
2. dbt (io.kestra.plugin.dbt.cli.DbtCLI) runs the project in a container using the io.kestra.plugin.scripts.runner.docker.Docker task runner with the ghcr.io/kestra-io/dbt-bigquery:latest image, so the BigQuery adapter and dependencies are preinstalled.
3. The service account JSON is injected as sa.json via inputFiles from the GCP_CREDS secret, and the inline profiles block points dbt at BigQuery using the service-account method, with location: EU, threads: 16, and a dev target.
4. The task executes dbt deps followed by dbt build, resolving packages and then running and testing models in one pass.

What you get

• Analytics-ready tables and models materialized in BigQuery from version-controlled dbt SQL.
• A containerized, dependency-isolated dbt run that behaves the same locally and in production.
• Secure service-account authentication with no credentials committed to the repository.
• Full task-level logs and run history for every dbt deps and dbt build invocation.

Who it's for

• Analytics engineers running dbt on BigQuery.
• Data platform teams standardizing reproducible ELT pipelines.
• Anyone moving dbt off laptops and brittle cron jobs onto managed orchestration.

Why orchestrate this with Kestra

dbt's own CLI can run models but cannot schedule, retry, or connect them to upstream events on its own. With Kestra you can attach event or schedule triggers, add automatic retries on transient BigQuery failures, capture per-task logs and lineage, and define the whole pipeline as declarative YAML kept under version control. This closes the gap dbt Core leaves open: turning a manual command into a governed, observable, production workflow.

Prerequisites

• A Google Cloud project with BigQuery enabled and a target dataset.
• A service account with permission to read and write the dataset.
• Docker available to the Kestra worker for the container task runner.

Secrets

• GCP_CREDS: the full Google Cloud service account JSON key, injected as sa.json for dbt's BigQuery profile.

Quick start

1. Add the GCP_CREDS secret with your service account JSON.
2. Update the profiles block with your BigQuery dataset, project, and location.
3. Point clone_repository at your own dbt project repository and branch.
4. Execute the flow and inspect the dbt task logs to confirm the build.

How to extend

• Add a io.kestra.plugin.core.trigger.Schedule trigger to run transformations on a cadence.
• Swap dbt build for dbt run, dbt test, or selectors like dbt build --select tag:daily.
• Parameterize the dataset, target, or branch with flow inputs for multiple environments.
• Chain downstream tasks to export results, notify Slack, or refresh dashboards.

Links

• Plugin dbt documentation
• dbt Core orchestration with Kestra
• BigQuery orchestration with Kestra