Arrow Flight Queries
CertifiedExecute multiple SQL statements using Apache Arrow Flight SQL
Arrow Flight Queries
Execute multiple SQL statements using Apache Arrow Flight SQL
Runs multiple SQL statements using Apache Arrow Flight SQL protocol for high-performance data transfer. Arrow Flight uses columnar data format and gRPC for efficient communication. Supports parameterized queries, transactions (default enabled), and all fetch modes. Default fetchSize is 10,000 rows.
type: io.kestra.plugin.jdbc.arrowflight.QueriesExamples
Send SQL queries to a database and fetch row(s) using Apache Arrow Flight SQL driver.
id: arrow_flight_sql_queries
namespace: company.team
tasks:
- id: query
type: io.kestra.plugin.jdbc.arrowflight.Queries
url: jdbc:arrow-flight-sql://localhost:31010/?useEncryption=false
username: "{{ secret('ARROWFLIGHT_USERNAME') }}"
password: "{{ secret('ARROWFLIGHT_PASSWORD') }}"
sql: SELECT * FROM employee; SELECT * FROM laptop;
fetchType: FETCH
Properties
sql *Requiredstring
SQL statement(s) to execute
Runs one or more SQL statements rendered with flow variables. Query tasks accept a single statement; Queries tasks can execute multiple statements separated by semicolons
url *Requiredstring
The JDBC URL to connect to the database
afterSQL string
SQL to execute after main query in same transaction
Optional SQL executed in the same transaction after the main statement. Useful for marking rows as processed to avoid duplicates; only a single statement is allowed. Commit covers both sql and afterSQL
connectionPoolSize integerstring
10Maximum number of pooled connections
Maximum connections held in the pool for a given URL and credentials. Default 10. Increase for flows that run many concurrent queries against the same database to avoid waiting for an available connection. Ignored when connectionPooling is false or for embedded drivers.
connectionPooling booleanstring
trueReuse database connections via a connection pool
When true (default), connections are pooled and reused across executions, keyed by URL and credentials, removing the connect and TLS-handshake cost on each run. Set to false if your SQL relies on session state persisting on the connection (for example SET search_path, session-scoped temp tables or variables), since pooled connections are reused. Embedded drivers (DuckDB, SQLite, MS Access) never pool regardless of this setting.
fetchSize integerstring
10000Number of rows to fetch per database round trip
Controls JDBC fetch size for STORE mode. Default: 10,000 rows; use Integer.MIN_VALUE for MySQL streaming. Ignored for FETCH and FETCH_ONE
fetchType string
NONESTOREFETCHFETCH_ONENONEResult fetching mode
FETCH returns all rows, FETCH_ONE returns the first row only, STORE streams rows to internal storage (ION), NONE returns no data. Default: NONE
outputFiles array
Output file names to capture after SQL execution
Creates named temporary files in the task working directory before the SQL runs, making their absolute paths available as {{ outputFiles.name }} Pebble variables in the SQL template. Only supported by embedded, in-process drivers (DuckDB, SQLite) where the database engine writes to the same filesystem as the Kestra worker. Remote database drivers (Postgres, MySQL, etc.) do not support this — they execute SQL on a separate server that cannot write to the Kestra worker filesystem.
parameters object
Named parameter bindings for SQL query
Map of parameter names to values. Use : name placeholders rendered then bound as prepared-statement parameters; supports nulls and typed values
password string
The database user's password
pluginDefaultsRef Non-dynamicstring
Reference (ref) of the pluginDefaults to apply to this task.
timeZoneId string
The time zone id to use for date/time manipulation. Default value is the worker's default time zone id
transaction booleanstring
trueTransaction
If one query failed, rollback transactions.
username string
The database user
Outputs
outputs array
The list of per-query outputs
io.kestra.plugin.jdbc.AbstractJdbcBaseQuery-Output
First row of fetched data
Only populated when fetchType is FETCH_ONE
List of all fetched rows
Only populated when fetchType is FETCH
Number of rows fetched
Only populated when fetchType is FETCH or STORE
uriURI of stored results in internal storage
Only populated when fetchType is STORE; file is stored in internal storage using ION format
Metrics
fetch.size counter
rowsThe number of fetched rows.
