Queries
Run multiple DuckDB queries.
type: "io.kestra.plugin.jdbc.duckdb.Queries"Examples
Execute multiple queries that reads a csv, and outputs a select and a count.
id: queries_duckdb
namespace: company.team
tasks:
- id: http_download
type: io.kestra.plugin.core.http.Download
uri: "https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv"
- id: queries
type: io.kestra.plugin.jdbc.duckdb.Queries
url: 'jdbc:duckdb:'
timeZoneId: Europe/Paris
sql: |-
CREATE TABLE new_tbl AS SELECT * FROM read_csv_auto('in.csv', header=True);
SELECT count(customer_name) FROM new_tbl;
SELECT customer_name FROM new_tbl;
inputFiles:
in.csv: "{{ outputs.http_download.uri }}"
Execute a query that reads a CSV file, and outputs another CSV file.
id: query_duckdb
namespace: company.team
tasks:
- id: http_download
type: io.kestra.plugin.core.http.Download
uri: "https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv"
- id: query
type: io.kestra.plugin.jdbc.duckdb.Queries
url: 'jdbc:duckdb:'
timeZoneId: Europe/Paris
sql: |-
CREATE TABLE new_tbl AS SELECT * FROM read_csv_auto('data.csv', header=True);
COPY (SELECT order_id, customer_name FROM new_tbl) TO '{{ outputFiles.out }}' (HEADER, DELIMITER ',');
inputFiles:
data.csv: "{{ outputs.http_download.uri }}"
outputFiles:
- out
Properties
afterSQLstring
SQL to execute atomically after trigger query.
Optional SQL executed in the same transaction as the main trigger query. Typically updates processing flags to prevent duplicate processing. Both sql and afterSQL queries commit together, ensuring consistency.
databaseUristring
Database URI
Kestra's URI to an existing Duck DB database file
fetchSizeintegerstring
10000Number of rows that should be fetched.
Gives the JDBC driver a hint as to the number of rows that should be fetched from the database when more rows are needed for this ResultSet object. If the fetch size specified is zero, the JDBC driver ignores the value and is free to make its own best guess as to what the fetch size should be. Ignored if autoCommit is false.
fetchTypestring
NONESTOREFETCHFETCH_ONENONEThe way you want to store data.
FETCH_ONE - output the first row. FETCH - output all rows as output variable. STORE - store all rows to a file. NONE - do nothing.
inputFilesobject
Input files to be loaded from DuckDb.
Describe a files map that will be written and usable by DuckDb. You can reach files by their filename, example: SELECT * FROM read_csv_auto('myfile.csv');
outputDbFilebooleanstring
falseOutput the database file.
This property lets you define if you want to output the in-memory database as a file for further processing.
outputFilesarray
Output file list that will be uploaded to internal storage.
List of keys that will generate temporary files.
On the SQL query, you can just use a variable named outputFiles.key for the corresponding file.
If you add a file with ["first"], you can use the special vars COPY tbl TO '{{ outputFiles.first }}' (HEADER, DELIMITER ','); and use this file in others tasks using {{ outputs.taskId.outputFiles.first }}.
parametersobject
Parameters
A map of parameters to bind to the SQL queries. The keys should match the parameter placeholders in the SQL string, e.g., : parameterName.
passwordstring
The database user's password.
sqlstring
SQL statement(s) to execute.
Runs one or more SQL statements depending on the task type. Query tasks support a single SQL statement, while Queries tasks can run multiple statements separated by semicolons.
timeZoneIdstring
The time zone id to use for date/time manipulation. Default value is the worker's default time zone id.
transactionbooleanstring
trueTransaction
If one query failed, rollback transactions.
urlstring
jdbc:duckdb:The JDBC URL to connect to the database.
The default value, jdbc: duckdb: , will use a local in-memory database.
Set this property when connecting to a persisted database instance, for example jdbc: duckdb: md: my_database?motherduck_token=<my_token> to connect to MotherDuck.
usernamestring
The database user.
Outputs
outputsarray
io.kestra.plugin.jdbc.AbstractJdbcBaseQuery-Output
Map containing the first row of fetched data.
Only populated if fetchOne parameter is set to true.
List of map containing rows of fetched data.
Only populated if fetch parameter is set to true.
The number of rows fetched.
Only populated if store or fetch parameter is set to true.
uriThe URI of the result file on Kestra's internal storage (.ion file / Amazon Ion formatted text file).
Only populated if store is set to true.
Metrics
fetch.sizecounter
rowsThe number of fetched rows.
