Query
type: "io.kestra.plugin.jdbc.duckdb.Query"
Query a local DuckDb
Examples
Execute a query that read a csv and output another one
id: "query"
type: "io.kestra.plugin.jdbc.duckdb.Query"
url: 'jdbc:duckdb:'
timeZoneId: Europe/Paris
sql: |-
CREATE TABLE new_tbl AS SELECT * FROM read_csv_auto('{{workingDir}}/in.csv', header=True);
COPY (SELECT id, name FROM new_tbl) TO '{{ outputFiles.out }}' (HEADER, DELIMITER ',');
inputFiles:
in.csv: {{ inputs.csv }}
outputFiles:
- out
Properties
url
- Type: string
- Dynamic: ✔️
- Required: ✔️
- Default:
jdbc:duckdb:null
The jdbc url to connect to the database
autoCommit
- Type: boolean
- Dynamic: ❌
- Required: ❌
- Default:
true
If autocommit is enabled
Sets this connection's auto-commit mode to the given state. If a connection is in auto-commit mode, then all its SQL statements will be executed and committed as individual transactions. Otherwise, its SQL statements are grouped into transactions that are terminated by a call to either the method commitor the method rollback. By default, new connections are in auto-commit mode except if you are using a store
properties that will disabled autocommit whenever this properties values.
fetch
- Type: boolean
- Dynamic: ❌
- Required: ❌
- Default:
false
Whether to fetch the data from the query result to the task output This parameter is evaluated after 'fetchOne' and 'store'.
fetchOne
- Type: boolean
- Dynamic: ❌
- Required: ❌
- Default:
false
Whether to fetch only one data row from the query result to the task output. This parameter is evaluated before 'store' and 'fetch'.
fetchSize
- Type: integer
- Dynamic: ❌
- Required: ❌
- Default:
10000
Number of rows that should be fetched
Gives the JDBC driver a hint as to the number of rows that should be fetched from the database when more rows are needed for this ResultSet object. If the fetch size specified is zero, the JDBC driver ignores the value and is free to make its own best guess as to what the fetch size should be. Ignored if autoCommit
is false.
inputFiles
- Type: object
- SubType: string
- Dynamic: ✔️
- Required: ❌
Input files to be loaded from DuckDb.
Describe a files map that will be written and usable by DuckDb. You can reach files using a workingDir
variable, example: SELECT * FROM read_csv_auto('{{workingDir}}/myfile.csv');
outputFiles
- Type: array
- SubType: string
- Dynamic: ❌
- Required: ❌
Output file list that will be uploaded to internal storage
List of key that will generate temporary files.
On the sql query, just can use with special variable named outputFiles.key
.
If you add a files with ["first"]
, you can use the special vars COPY tbl TO '{[ outputFiles.first }}' (HEADER, DELIMITER ',');
and you used on others tasks using null
password
- Type: string
- Dynamic: ✔️
- Required: ❌
The database user's password
sql
- Type: string
- Dynamic: ✔️
- Required: ❌
The sql query to run
store
- Type: boolean
- Dynamic: ❌
- Required: ❌
- Default:
false
Whether to fetch data row from the query result to a file in internal storage. File will be saved as Amazon Ion (text format). See Amazon Ion documentation This parameter is evaluated after 'fetchOne' but before 'fetch'.
timeZoneId
- Type: string
- Dynamic: ❌
- Required: ❌
The time zone id to use for date/time manipulation. Default value is the worker default zone id.
username
- Type: string
- Dynamic: ✔️
- Required: ❌
The database user
Outputs
outputFiles
- Type: object
- SubType: string
The output files uri in Kestra internal storage
row
- Type: object
Map containing the first row of fetched data
Only populated if 'fetchOne' parameter is set to true.
rows
- Type: array
- SubType: object
Lit of map containing rows of fetched data
Only populated if 'fetch' parameter is set to true.
size
- Type: integer
The size of the fetched rows
Only populated if 'store' or 'fetch' parameter is set to true.
uri
- Type: string
The url of the result file on kestra storage (.ion file / Amazon Ion text format)
Only populated if 'store' is set to true.