Run dbt ELT pipelines on MotherDuck from Git using Docker

Run a complete dbt transformation pipeline against MotherDuck, the cloud-native DuckDB service, straight from a Git repository. This blueprint clones your dbt project, installs packages, and runs dbt build inside a pinned Docker image so models, tests, snapshots, and seeds all execute the same way locally, in CI, and in production. It solves the common problem of gluing together version-controlled dbt code, a serverless DuckDB warehouse, and a scheduler without writing custom shell scripts or maintaining a dbt runner host.

How it works

1. The git task (io.kestra.plugin.core.flow.WorkingDirectory) creates an isolated working directory shared across the nested tasks.
2. clone_repository (io.kestra.plugin.git.Clone) checks out the main branch of a dbt project from GitHub into that directory.
3. dbt_build (io.kestra.plugin.dbt.cli.DbtCLI) runs on the io.kestra.plugin.scripts.runner.docker.Docker task runner using the ghcr.io/kestra-io/dbt-duckdb:latest image. An inline profiles block points the dev target at MotherDuck via the DuckDB connection string md:my_dbt_project?motherduck_token={{secret('MOTHERDUCK_TOKEN')}}, with disable_transactions, threads: 16, and a timeout_seconds of 300.
4. The task runs dbt deps to install packages, then dbt build to materialize and test the project in one pass.

What you get

• A one-pass dbt build (run plus test plus snapshot plus seed) against MotherDuck.
• A reproducible Docker execution environment with dbt and the DuckDB adapter preinstalled.
• Git as the single source of truth for transformation logic.
• Token-based MotherDuck authentication kept out of the codebase.

Who it's for

• Analytics engineers running dbt on a serverless DuckDB warehouse.
• Data teams moving from local DuckDB to a shared, collaborative MotherDuck backend.
• Anyone needing lightweight ELT without standing up BigQuery, Snowflake, or Redshift.

Why orchestrate this with Kestra

dbt and the DuckDB CLI have no scheduler, no retry policy, and no native lineage across pipelines. Kestra adds event and schedule triggers, automatic retries on transient failures, run-level observability, and a declarative YAML definition that lives next to the rest of your stack. The Docker task runner pins the execution environment so a dbt build is identical everywhere, something a bare dbt invocation cannot guarantee.

Prerequisites

• A MotherDuck account and a service token.
• A dbt project in Git (the example uses https://github.com/kestra-io/dbt-example).
• Docker available to the Kestra worker for the task runner.

Secrets

• MOTHERDUCK_TOKEN: the MotherDuck service token used in the DuckDB connection string.

Quick start

1. Add MOTHERDUCK_TOKEN as a secret in your Kestra instance.
2. Point clone_repository at your own dbt project URL and branch.
3. Update the profiles block so the project name and target match your dbt project.
4. Execute the flow and inspect the dbt logs in the Kestra UI.

How to extend

• Add a io.kestra.plugin.core.trigger.Schedule trigger to run the pipeline on a cadence.
• Split dbt build into separate dbt run and dbt test commands for finer control.
• Parameterize the branch or target with flow inputs to support CI and production environments.
• Add downstream tasks to publish dbt artifacts or notify a channel on completion.

Links

• plugin-dbt documentation
• dbt Core orchestration with Kestra