Duckdb Queries

Duckdb Queries

Certified

Execute multiple SQL statements against DuckDB

Runs multiple SQL statements in DuckDB embedded analytical database. Supports reading from CSV, Parquet, and JSON files directly. Can work with in-memory databases or persistent database files. Supports parameterized queries, transactions (default enabled), and all fetch modes. Can output the database file to internal storage.

yaml
type: io.kestra.plugin.jdbc.duckdb.Queries

Execute queries that export results to a file, which is automatically captured as an output file.

yaml
id: queries_export
namespace: company.team

tasks:
  - id: queries
    type: io.kestra.plugin.jdbc.duckdb.Queries
    sql: |-
      CREATE TABLE users (id INT, name VARCHAR);
      INSERT INTO users VALUES (1, 'John'), (2, 'Jane');
      COPY users TO '{{ workingDir }}/users_export.csv' (HEADER, DELIMITER ',');

Execute multiple queries that reads a csv, and outputs a select and a count.

yaml
id: queries_duckdb
namespace: company.team

tasks:
  - id: http_download
    type: io.kestra.plugin.core.http.Download
    uri: "https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv"

  - id: queries
    type: io.kestra.plugin.jdbc.duckdb.Queries
    url: 'jdbc:duckdb:'
    timeZoneId: Europe/Paris
    sql: |-
      CREATE TABLE new_tbl AS SELECT * FROM read_csv_auto('in.csv', header=True);
      SELECT count(customer_name) FROM new_tbl;
      SELECT customer_name FROM new_tbl;
    inputFiles:
      in.csv: "{{ outputs.http_download.uri }}"

Execute a query that reads a CSV file, and outputs another CSV file.

yaml
id: query_duckdb
namespace: company.team

tasks:
  - id: http_download
    type: io.kestra.plugin.core.http.Download
    uri: "https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv"

  - id: query
    type: io.kestra.plugin.jdbc.duckdb.Queries
    url: 'jdbc:duckdb:'
    timeZoneId: Europe/Paris
    sql: |-
      CREATE TABLE new_tbl AS SELECT * FROM read_csv_auto('data.csv', header=True);

      COPY (SELECT order_id, customer_name FROM new_tbl) TO '{{ outputFiles.out }}' (HEADER, DELIMITER ',');
    inputFiles:
      data.csv: "{{ outputs.http_download.uri }}"
    outputFiles:
       - out
Installation

Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command: 

kestractl plugins get io.kestra.plugin:plugin-jdbc-duckdb --compatible-for $KESTRA_VERSION

For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.

For Kestra CLI, replace LATEST with a specific version or version range to pin the plugin version.

Properties

SQL statement(s) to execute

Runs one or more SQL statements rendered with flow variables. Query tasks accept a single statement; Queries tasks can execute multiple statements separated by semicolons

SQL to execute after main query in same transaction

Optional SQL executed in the same transaction after the main statement. Useful for marking rows as processed to avoid duplicates; only a single statement is allowed. Commit covers both sql and afterSQL

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions
assetFailureBehaviorstring
Possible Values
IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

id*string
Min length1
typestring
outputs

The assets produced as outputs.

id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*string
Min length1

Custom asset type

descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
Defaulttrue

Whether to automatically capture output files

Whether to automatically capture and upload files created in the working directory during execution. Defaults to true. Note that files ending in .db, .wal, or .tmp, the task's database file, the .duckdb_extensions/ directory, and pre-existing unchanged files are excluded from automatic capture.

Defaultfalse

Compute per-column statistics of the fetched rows

When true, writes an ION file with one entry per column (column, jdbcType, nulls, distinct (approximate), min, max, mean, median, minLen, maxLen) and exposes its URI in the columnStats output. Statistics are computed while the rows are read, without running a second query. Only applies to fetchType FETCH and STORE; LOB, binary and array columns are skipped. min and max are omitted for string columns longer than 256 characters, and median is computed on a bounded sample of the values

Default100000

Number of rows used to compute column statistics

Only used when columnStats is true. Statistics stop at this many rows while all rows are still fetched or stored. Must be at least 1. Default: 100,000

SubTypestring
Default["ion"]

DuckDB community extensions to install and load before running the SQL

Defaults to ["ion"]. Each extension is attempted on a best-effort basis using INSTALL <ext> FROM community followed by LOAD <ext>. If installation or loading fails, Kestra logs a warning and continues.

Database URI

Kestra's URI to an existing Duck DB database file

Default10000

Number of rows to fetch per database round trip

Controls JDBC fetch size for STORE mode. Default: 10,000 rows; use Integer.MIN_VALUE for MySQL streaming. Ignored for FETCH and FETCH_ONE

DefaultNONE
Possible Values
STOREFETCHFETCH_ONENONE

Result fetching mode

FETCH returns all rows, FETCH_ONE returns the first row only, STORE streams rows to internal storage (ION), NONE returns no data. Default: NONE

Input files to be loaded from DuckDb

Describe a files map that will be written and usable by DuckDb. You can reach files by their filename, example: SELECT * FROM read_csv_auto('myfile.csv'); Relative paths are resolved by DuckDB against the Kestra process directory, not the task working directory; the working directory is only used as a fallback for reads when no file with that name exists in the process directory. If such a file exists, the task fails to avoid silently reading the wrong data. To be unambiguous, reference the file as '{{ workingDir }}/myfile.csv'.

Default104857600

Maximum total size (in bytes) of output files to automatically capture

Maximum combined size in bytes of new or modified files to automatically capture and upload. If exceeded, the task will fail. Defaults to 104857600 (100 MB).

Default100

Maximum number of output files to automatically capture

Maximum number of new or modified files to automatically capture and upload. If exceeded, the task will fail. Defaults to 100.

Defaultfalse

Output the database file

This property lets you define if you want to output the in-memory database as a file for further processing.

Defaultfalse

Expose the executed SQL statement in the outputs

When true, the rendered SQL is exposed in the statement output, with the values of the secret() calls used in sql replaced by *** and truncated to 64 KiB. Disabled by default so that SQL text never leaks into outputs unless requested

Named parameter bindings for SQL query

Map of parameter names to values, bound to : name placeholders as prepared-statement parameters; supports nulls and typed values. Colons in literals, quoted identifiers, comments and :: casts are left as is; a : name missing from this map is bound to NULL with a warning.

The database user's password

The time zone id to use for date/time manipulation. Default value is the worker's default time zone id

Defaulttrue

Transaction

If one query failed, rollback transactions.

Defaultjdbc:duckdb:

The JDBC URL to connect to the database

The default value, jdbc: duckdb: , will use a local in-memory database. Set this property when connecting to a persisted database instance, for example jdbc: duckdb: md: my_database?motherduck_token=<my_token> to connect to MotherDuck.

The database user

SubTypeinteger

The number of affected rows for each query that doesn't return a result set

One entry per INSERT, UPDATE, DELETE or DDL statement, in execution order, as reported by the JDBC driver (DDL statements usually report 0). Queries returning a result set are not included here; their results are in outputs.

Total size of the stored result files in bytes

Only populated when fetchType is STORE

Formaturi

URI of the column statistics file in internal storage

Only populated when columnStats is true and fetchType is FETCH or STORE; columns of all result sets are merged by name

Total execution duration in milliseconds

Includes reading the result rows of every statement

The list of per-query outputs

Definitions
bytesinteger

Size of the stored result file in bytes

Only populated when fetchType is STORE

columnStatsstring
Formaturi

URI of the column statistics file in internal storage

Only populated when columnStats is true and fetchType is FETCH or STORE; file is stored in ION format

durationMsinteger

Statement execution duration in milliseconds

Includes reading the result rows

queryIdstring

Engine query identifier

The id the engine coordinator assigned to the query, for example the Trino query id. Not set when the engine does not expose one.

rowobject

First row of fetched data

Only populated when fetchType is FETCH_ONE

rowsarray
SubTypeobject

List of all fetched rows

Only populated when fetchType is FETCH

rowsAffectedinteger

Number of rows affected by the statement

Reported by the JDBC driver for INSERT, UPDATE, DELETE and similar statements. Not set when the statement returns a result set or when the driver does not report a count

sizeinteger

Number of rows fetched

Only populated when fetchType is FETCH or STORE

statementstring

Executed SQL statement

Only populated when outputStatement is true. Values of secret() calls are replaced by *** and the text is truncated to 64 KiB

tablesarray

Tables read or written by the statement

Detected from the SQL text, so this is a best effort: CTEs, temporary tables and dialect specific syntax can lower the confidence or be missed. Limited to 50 entries, see tablesTruncated

accessstring
Possible Values
READWRITECREATEDROP

How the statement uses the table

One of READ, WRITE, CREATE or DROP.

confidencestring
Possible Values
MEDIUMLOW

How reliable the detection is

MEDIUM when detected from the SQL text, LOW for temporary tables and ambiguous names.

databasestring

Database name

Null when the statement does not qualify the table with a database.

kindstring
Possible Values
TABLEFILEURL

Kind of source

Only set on DuckDB: TABLE, FILE or URL (files read through read_parquet, read_csv, ATTACH and quoted paths).

namestring

Table name

For DuckDB file reads, the file path or URL.

schemastring

Schema name

Null when the statement does not qualify the table with a schema.

tablesTruncatedboolean

Whether the tables list was cut at 50 entries

uristring
Formaturi

URI of stored results in internal storage

Only populated when fetchType is STORE; file is stored in internal storage using ION format

warningsinteger

Number of SQL warnings raised by the statement

Count of the SQLWarning chain reported by the driver, for example Postgres notices

Total number of rows affected by all statements

Sum of affectedRows. Not set when no statement reported an update count

Executed SQL statements

Only populated when outputStatement is true. Values of secret() calls are replaced by *** and the text is truncated to 64 KiB

Number of executed statements

When the driver runs the whole script in a single batch (transactional Snowflake, Redshift and Sybase scripts), the script counts as one statement

Formaturi

URI of the per-statement details file in internal storage

ION file with one {index, kind, rowsAffected, durationMs} entry per executed statement, plus queryId for engines that expose one (e.g. Trino).

Tables read or written by the statements

Detected from the SQL text, so this is a best effort: CTEs, temporary tables and dialect specific syntax can lower the confidence or be missed. Limited to 50 entries, see tablesTruncated

Definitions
accessstring
Possible Values
READWRITECREATEDROP

How the statement uses the table

One of READ, WRITE, CREATE or DROP.

confidencestring
Possible Values
MEDIUMLOW

How reliable the detection is

MEDIUM when detected from the SQL text, LOW for temporary tables and ambiguous names.

databasestring

Database name

Null when the statement does not qualify the table with a database.

kindstring
Possible Values
TABLEFILEURL

Kind of source

Only set on DuckDB: TABLE, FILE or URL (files read through read_parquet, read_csv, ATTACH and quoted paths).

namestring

Table name

For DuckDB file reads, the file path or URL.

schemastring

Schema name

Null when the statement does not qualify the table with a schema.

Whether the tables list was cut at 50 entries

Number of SQL warnings raised by the statements

Count of the SQLWarning chains reported by the driver across all statements

Unitbytes

The size of the stored result file.

The duration of the SQL execution, including reading the result rows.

Unitrows

The number of fetched rows.

Unitrows

The number of rows affected, as reported by the JDBC driver.

More Plugins in this Category

Firecrawl
Firecrawl

Tasks that scrape and crawl websites into clean, LLM-ready content with Firecrawl.

Data

2 tasks

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks

8 blueprints