Run dbt ELT pipelines with DuckDB from Git using Docker

Run a complete dbt ELT pipeline on DuckDB straight from a Git repository, inside a single Docker container, with no cloud data warehouse to provision. This blueprint clones a versioned dbt project, installs its package dependencies, and executes a full dbt build against an in-memory DuckDB database. It solves the classic dbt operating problem: dbt transforms data well but has no native scheduler, retry logic, or dependency orchestration, so teams end up wiring cron jobs and shell scripts around it. Here Kestra handles the orchestration declaratively while dbt and DuckDB do the modeling and compute.

How it works

The flow runs everything inside a single io.kestra.plugin.core.flow.WorkingDirectory task so all steps share one filesystem and one container session:

1. clone_repository uses io.kestra.plugin.git.Clone to check out the main branch of the kestra-io/dbt-demo repository into the working directory.
2. dbt_core uses io.kestra.plugin.dbt.cli.DbtCLI with a Docker taskRunner (io.kestra.plugin.scripts.runner.docker.Docker) and the ghcr.io/kestra-io/dbt-duckdb:latest image. An inputFiles entry injects packages.yml (pulling in dbt_artifacts 2.10.1), and an inline profiles block defines a DuckDB target using an in-memory path with the parquet extension.
3. The task runs dbt deps to install packages, then dbt build to compile, run, test, and materialize all models in one pass.

What you get

• A reproducible, version-controlled dbt run tied to a Git commit
• dbt deps and dbt build executed in one container to minimize startup latency
• In-memory DuckDB compute, so no external database or warehouse is required
• Captured dbt run artifacts via the dbt_artifacts package for downstream lineage
• A pattern you can drop into CI or schedule without rewriting your dbt project

Who it's for

• Analytics engineers who want orchestrated dbt without managing a scheduler
• Data teams prototyping models on DuckDB before promoting to Snowflake, BigQuery, or Redshift
• Platform engineers standardizing dbt CI and pull-request validation
• Anyone running local-first or embedded analytics workloads

Why orchestrate this with Kestra

dbt's own CLI cannot schedule itself, retry failed steps, react to events, or coordinate with upstream and downstream tasks. Kestra adds event and schedule triggers, automatic retries, full execution lineage, and declarative YAML you can version alongside the dbt project itself. You get one place to see when the build ran, why it failed, and what depended on it, instead of stitching that together from cron logs.

Prerequisites

• A Kestra instance with Docker available to the worker (the task runner pulls and runs ghcr.io/kestra-io/dbt-duckdb:latest)
• Network access to clone the public kestra-io/dbt-demo repository

Secrets

This blueprint clones a public repository and runs DuckDB in memory, so no Kestra secrets are required. If you point clone_repository at a private repo, add a credential such as {{ secret('GITHUB_TOKEN') }}.

Quick start

1. Add this flow to your Kestra instance.
2. Confirm the worker can run Docker containers.
3. Execute the flow and watch clone_repository then dbt_core in the Gantt and Logs views.
4. Inspect the dbt deps and dbt build output in the task logs.

How to extend

• Repoint clone_repository at your own dbt project and branch.
• Swap the DuckDB profile for Snowflake, BigQuery, or Postgres by editing the profiles block and image.
• Add a Schedule or Webhook trigger to run the build on a cadence or on push.
• Add dbt test or dbt source freshness commands, or split into separate tasks with retries.
• Persist DuckDB output to a file or object store instead of :memory: for durable results.

Links

• Git plugin documentation
• dbt plugin documentation
• dbt Core orchestration with Kestra
• DuckDB orchestration with Kestra