Getting started with Kestra, a Data Warehouse and Analytics workflow example

Run an end-to-end data warehouse and analytics pipeline that clones a dbt project from Git, builds and tests the models on DuckDB, and exports the transformed tables to CSV for downstream analysis. This blueprint is the fastest way to see dbt orchestration in action with Kestra, turning raw source data into clean, analysis-ready datasets in a single declarative workflow with no warehouse credentials or local setup required.

How it works

The flow runs three tasks inside one isolated io.kestra.plugin.core.flow.WorkingDirectory so every step shares the same files:

1. clone_repository uses io.kestra.plugin.git.Clone to pull the kestra-io/dbt-demo repository (the main branch) into the working directory.
2. dbt_build uses io.kestra.plugin.dbt.cli.DbtCLI on the ghcr.io/kestra-io/dbt-duckdb Docker image to run dbt deps and dbt build. The inline profiles block points dbt at a local DuckDB database (dbt.duckdb) with the parquet extension and the CORE engine.
3. python uses io.kestra.plugin.scripts.python.Script with duckdb and pandas to inspect the DuckDB schema, skip raw/stg staging tables, and export each remaining modeled table to a CSV file captured through outputFiles.

What you get

• A working dbt build (models plus tests) running on an ephemeral DuckDB warehouse.
• Clean, modeled tables exported as downloadable CSV artifacts.
• A reproducible, containerized run with no warehouse provisioning.
• A template you can repoint at your own dbt repository and database.

Who it's for

• Analytics engineers learning how to orchestrate dbt with Kestra.
• Data engineers prototyping a transformation-to-export pipeline.
• Teams evaluating dbt and DuckDB before wiring up a production warehouse.

Why orchestrate this with Kestra

dbt's own CLI runs models but does not schedule, retry, or connect the steps around it. Kestra wraps the clone, build, and export into one declarative YAML workflow with event and schedule triggers, automatic retries, and full execution lineage and logs across Git, dbt, DuckDB, and Python. You get reproducible containerized runs and captured output artifacts, the orchestration layer dbt alone cannot provide.

Prerequisites

• A running Kestra instance with Docker available to the task runner.
• Network access to GitHub and the ghcr.io/kestra-io/dbt-duckdb image.

Secrets

This demo uses a public repository and a local DuckDB file, so no secrets are required. When you adapt it to a private repo or real warehouse, add credentials with {{ secret('NAME') }}.

Quick start

1. Add this blueprint to your Kestra instance.
2. Confirm the task runner can pull Docker images.
3. Execute the flow.
4. Download the exported CSV files from the python task outputs.

How to extend

• Swap the url and branch in clone_repository to build your own dbt project.
• Replace the DuckDB profiles with Snowflake, BigQuery, or Postgres targets.
• Add a schedule or event trigger to refresh the warehouse automatically.
• Load the CSV outputs into object storage or a database instead of exporting locally.

Links

• Plugin: dbt
• Orchestrate dbt Core with Kestra