Query
type: "io.kestra.plugin.jdbc.duckdb.Query"
Query a DuckDb Database.
Examples
Execute a query that reads a csv, and outputs another csv.
id: query_duckdb
namespace: company.team
tasks:
- id: http_download
type: io.kestra.plugin.core.http.Download
uri: "https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv"
- id: query
type: io.kestra.plugin.jdbc.duckdb.Query
url: 'jdbc:duckdb:'
timeZoneId: Europe/Paris
sql: |-
CREATE TABLE new_tbl AS SELECT * FROM read_csv_auto('{{ workingDir }}/in.csv', header=True);
COPY (SELECT order_id, customer_name FROM new_tbl) TO '{{ outputFiles.out }}' (HEADER, DELIMITER ',');
inputFiles:
in.csv: "{{ outputs.http_download.uri }}"
outputFiles:
- out
Execute a query that reads from an existing database file using a URL.
id: query_duckdb
namespace: company.team
tasks:
- id: query1
type: io.kestra.plugin.jdbc.duckdb.Query
url: jdbc:duckdb:/{{ vars.dbfile }}
sql: SELECT * FROM table_name;
fetchType: STORE
- id: query2
type: io.kestra.plugin.jdbc.duckdb.Query
url: jdbc:duckdb:/temp/folder/duck.db
sql: SELECT * FROM table_name;
fetchType: STORE
Execute a query that reads from an existing database file using the
databaseFile
variable.
id: query_duckdb
namespace: company.team
tasks:
- id: query1
type: io.kestra.plugin.jdbc.duckdb.Query
url: jdbc:duckdb:
databaseFile: {{ vars.dbfile }}
sql: SELECT * FROM table_name;
fetchType: STORE
- id: query2
type: io.kestra.plugin.jdbc.duckdb.Query
url: jdbc:duckdb:
databaseFile: /temp/folder/duck.db
sql: SELECT * FROM table_name;
fetchType: STORE
Properties
fetchType
- Type: string
- Dynamic: ❌
- Required: ✔️
- Default:
STORE
- Possible Values:
STORE
FETCH
FETCH_ONE
NONE
The way you want to store data.
FETCH_ONE - output the first row. FETCH - output all rows as output variable. STORE - store all rows to a file. NONE - do nothing.
url
- Type: string
- Dynamic: ✔️
- Required: ✔️
- Default:
jdbc:duckdb:
The JDBC URL to connect to the database.
The default value,
jdbc:duckdb:
, will use a local in-memory database. Set this property when connecting to a persisted database instance, for examplejdbc:duckdb:md:my_database?motherduck_token=<my_token>
to connect to MotherDuck.
autoCommit
- Type: boolean
- Dynamic: ❌
- Required: ❌
- Default:
true
Whether autocommit is enabled.
Sets this connection's auto-commit mode to the given state. If a connection is in auto-commit mode, then all its SQL statements will be executed and committed as individual transactions. Otherwise, its SQL statements are grouped into transactions that are terminated by a call to either the method commit or the method rollback. By default, new connections are in auto-commit mode except when you are using
store
property in which case the auto-commit will be disabled.
fetch
⚠ Deprecated
- Type: boolean
- Dynamic: ❌
- Required: ❌
- Default:
false
DEPRECATED, please use fetchType: FETCH
instead.
Whether to fetch the data from the query result to the task output. This parameter is evaluated after fetchOne
and store
.
fetchOne
⚠ Deprecated
- Type: boolean
- Dynamic: ❌
- Required: ❌
- Default:
false
DEPRECATED, please use fetchType: FETCH_ONE
instead.
Whether to fetch only one data row from the query result to the task output. This parameter is evaluated before store
and fetch
.
fetchSize
- Type: integer
- Dynamic: ❌
- Required: ❌
- Default:
10000
Number of rows that should be fetched.
Gives the JDBC driver a hint as to the number of rows that should be fetched from the database when more rows are needed for this ResultSet object. If the fetch size specified is zero, the JDBC driver ignores the value and is free to make its own best guess as to what the fetch size should be. Ignored if
autoCommit
is false.
inputFiles
- Type: object
- SubType: string
- Dynamic: ✔️
- Required: ❌
Input files to be loaded from DuckDb.
Describe a files map that will be written and usable by DuckDb. You can reach files using a
workingDir
variable, example:SELECT * FROM read_csv_auto('{{ workingDir }}/myfile.csv');
outputFiles
- Type: array
- SubType: string
- Dynamic: ❌
- Required: ❌
Output file list that will be uploaded to internal storage.
List of keys that will generate temporary files. On the SQL query, you can just use a variable named
outputFiles.key
for the corresponding file. If you add a file with["first"]
, you can use the special varsCOPY tbl TO '{{ outputFiles.first }}' (HEADER, DELIMITER ',');
and use this file in others tasks using{{ outputs.taskId.outputFiles.first }}
.
password
- Type: string
- Dynamic: ✔️
- Required: ❌
The database user's password.
sql
- Type: string
- Dynamic: ✔️
- Required: ❌
The SQL query to run.
store
⚠ Deprecated
- Type: boolean
- Dynamic: ❌
- Required: ❌
- Default:
true
DEPRECATED, please use fetchType: FETCH_STORE
instead.
Whether to fetch data row(s) from the query result to a file in internal storage. File will be saved as Amazon Ion (text format).
See Amazon Ion documentation This parameter is evaluated after fetchOne
but before fetch
.
timeZoneId
- Type: string
- Dynamic: ❌
- Required: ❌
The time zone id to use for date/time manipulation. Default value is the worker's default time zone id.
username
- Type: string
- Dynamic: ✔️
- Required: ❌
The database user.
Outputs
outputFiles
- Type: object
- SubType: string
- Required: ❌
The output files' URI in Kestra's internal storage.
row
- Type: object
- Required: ❌
Map containing the first row of fetched data.
Only populated if
fetchOne
parameter is set to true.
rows
- Type: array
- SubType: object
- Required: ❌
List of map containing rows of fetched data.
Only populated if
fetch
parameter is set to true.
size
- Type: integer
- Required: ❌
The number of rows fetched.
Only populated if
store
orfetch
parameter is set to true.
uri
- Type: string
- Required: ❌
- Format:
uri
The URI of the result file on Kestra's internal storage (.ion file / Amazon Ion formatted text file).
Only populated if
store
is set to true.
Was this page helpful?