DuckDB runs the SQL in-process, fast and local. Kestra gives it what an in-process engine cannot: a schedule, retries, multi-step pipelines, and somewhere to persist the result. Query files in place, productionize a notebook ETL, and run dbt on DuckDB, all with one execution history.
DuckDB is an in-process engine with no scheduler and no persistence beyond the process. Kestra wraps it with everything production needs: it queries a remote CSV or S3 Parquet file in place, schedules the run, persists the result and even the .duckdb file to storage, and chains DuckDB with Python, dbt, and Slack in one flow. The throwaway notebook session becomes a scheduled, observable job.
DuckDB runs the SQL, the file reads, and the joins. Kestra runs the steps around the query: what triggers it, what feeds it, where the result is persisted, and how it scales out.
DuckDB cannot trigger itself; it runs once, in a process, then exits. Kestra drives io.kestra.plugin.jdbc.duckdb.Query on a cron Schedule, on a file arrival, or on a webhook, with per-run retries and backfills. The analysis you ran by hand becomes a job that runs on time, every time, and alerts you when it does not.
DuckDB reads CSV, Parquet, and JSON directly from a URL or object store. Kestra's inputFiles writes the file into the task working directory so the SQL reads it by name, and fetchType: STORE streams the result to internal storage. You get warehouse-style analytics over a lake with zero infrastructure to provision.
A default in-memory DuckDB vanishes when the process ends. Kestra's outputFiles and outputDbFile capture query results and the whole .duckdb file into internal storage, so the next task or flow reads them. The in-process engine gains durable, shareable state without running a server.
Real work is download, transform, query, serialize, notify. DuckDB does one SQL step. Kestra sequences it with core.http.Download, a Python or Polars transform, a serdes conversion, and a notification, with outputs flowing between tasks. The SQL becomes one node in a real DAG, not the whole script.
DuckDB is the fastest way to run dbt or SQLMesh models in CI, with no warehouse credentials. Kestra clones the project with git.Clone, runs dbt build or SQLMeshCLI against DuckDB in a container, then queries the result. Prototype locally, then point the same flow at Snowflake for production.
Local DuckDB is perfect until the data or the team outgrows one process. The same Query task points at MotherDuck by changing the JDBC URL to jdbc:duckdb:md: with a token, so a flow that prototyped on a laptop runs against a shared cloud database with no rewrite of the SQL.
Patterns analytics engineering teams run in production today. Each one shows the flow end to end, with the real plugin classes in play.
A dataset lives behind a URL or in object storage. Kestra downloads it, runs a DuckDB Query with inputFiles so the SQL reads it by name, then exports the result as CSV. Warehouse-style aggregation over a flat file, with nothing to provision and the result streamed to storage.
inputFiles drops the file in the working dir so read_csv_auto just works.
fetchType: STORE writes rows to internal storage as ION for the next task.
No warehouse, no credentials, no cluster. DuckDB runs in the task process.
Business metrics live in a CSV or object store, and someone keeps checking them by hand. Kestra runs a DuckDB Queries aggregation on a cron Schedule, formats the numbers, and posts a summary to Slack every Monday, replacing the manual spreadsheet check with a job.
A cron Schedule runs the report without anyone opening a notebook.
INSTALL httpfs; LOAD httpfs; lets DuckDB query the file straight from a URL.
The summary lands in the channel, not in a file someone has to open.
An analyst's notebook pulls an API, shapes it in Polars, and queries it in DuckDB. Kestra runs the same three steps as a flow: http.Request fetches the data, a Python step transforms it with outputFiles, then a DuckDB Query aggregates it, on a schedule with retries.
The notebook's fetch, transform, and query become three ordered, retryable tasks.
Python writes a file via outputFiles that DuckDB reads with inputFiles.
Each step runs in a pinned container, so the run is reproducible anywhere.
Transformation projects live in version control. Kestra clones the repo, runs SQLMeshCLI (or dbt build) against a DuckDB engine in a container, then queries the materialized result. A fast, credential-free CI engine for models you later ship to a cloud warehouse.
DuckDB runs the models in-process, so pull-request checks need no warehouse.
Swap SQLMeshCLI for dbt.cli.DbtCLI; the engine stays DuckDB.
Point the same flow at BigQuery or Snowflake for production runs.
A slow-changing reference table is downloaded on every run, wasting time and load. Kestra caches that download for 24 hours with taskCache, pulls only the fresh transactional data each run, then joins both in DuckDB, so the heavy fetch happens once a day, not every execution.
The reference download is reused for 24 hours, cutting load on the source.
Only the transactional pull runs every execution; the join sees both.
The expensive fetch happens once a day instead of on every run.
| Capability | ||||
|---|---|---|---|---|
| Schedule and trigger DuckDB runs | Cron, event, and file triggers | Schedules + sensors, Python | Schedules, Python flows | Cron schedules |
| Query files in place (CSV, Parquet, S3) | inputFiles + DuckDB SQL | Custom asset code | Custom task code | Custom script |
| Persist the in-memory .duckdb file | outputFiles + outputDbFile | Manual IO manager | Manual artifact code | Manual |
| Run dbt or SQLMesh on DuckDB | Native dbt + SQLMesh tasks | dbt integration, Python | prefect-dbt, Python | Shell step |
| Declarative YAML, no Python required | Pure YAML | Python required | Python required | TypeScript or Python scripts |
| Scale local DuckDB to MotherDuck | Change the JDBC URL | Custom code | Custom code | Custom script |
| Retries, backfills, replay from failure | Per-task, built in | Yes, Python config | Yes, Python config | Limited |
Find answers to your questions right here, and don't hesitate to Contact Us if you couldn't find what you're looking for.
Query files in place, schedule and persist runs, productionize notebook ETLs, and run dbt on a DuckDB engine in one flow. Open source, self-hosted, event-driven.
