Database
How to configure Kestra to use a database for its Queue and Repository.
First, you need to configure Kestra to use a database for its Queue and Repository. Here is an example for PostgreSQL:
kestra:
queue:
type: postgres
repository:
type: postgres
Currently, Kestra supports Postgres, H2, MySQL, and SQL Server (available in a preview in the Enterprise Edition since Kestra 0.18.0).
- H2 can be convenient for local development.
- For production, we recommend PostgreSQL. If PostgreSQL is not an option for you, MySQL and SQL Server can be used as well.
Check the Software Requirements section for the minimum version of each database.
If you experience performance issues when using PostgreSQL, you can tune the cost optimizer parameter random_page_cost=1.1
, which should make PostgreSQL use the right index for the queues table. You can also configure kestra.queue.postgres.disable-seq-scan=true
so that Kestra turns off sequential scans on the queue polling query forcing PostgreSQL to use the index.
Minimal configuration
In order to connect to a database, you need to configure a datasource
. Kestra uses The HikariCP connection pool under the hood, and if needed, you can configure multiple options from the HikariCP documentation directly in your datasource configuration.
PostgreSQL
Here is a minimal configuration for PostgreSQL:
kestra:
queue:
type: postgres
repository:
type: postgres
datasources:
postgres:
url: jdbc:postgresql://localhost:5432/kestra
driverClassName: org.postgresql.Driver
username: kestra
password: k3str4
MySQL
Here is a minimal configuration for MySQL:
kestra:
queue:
type: mysql
repository:
type: mysql
datasources:
mysql:
url: jdbc:mysql://localhost:3306/kestra
driverClassName: com.mysql.cj.jdbc.Driver
username: kestra
password: k3str4
dialect: MYSQL
Note that we currently don't support the 8.0.31
version of MySQL. If possible, try another version or reach out via Slack to help you troubleshoot.
SQL Server
Here is a minimal configuration for a SQL Server database, currently available in preview in the Enterprise Edition since Kestra 0.18.0:
kestra:
repository:
type: sqlserver
queue:
type: sqlserver
datasources:
sqlserver:
url: jdbc:sqlserver://localhost:1433
driverClassName: com.microsoft.sqlserver.jdbc.SQLServerDriver
username: kestra
password: k3str4
dialect: SQL_SERVER
# Uncomment only if your database is not encrypted
# data-source-properties:
# encrypt: false
Note that this feature is currently in preview and only available in the Enterprise Edition. If you want to try it, please reach out to get access.
H2
Here is a minimal configuration for H2:
kestra:
queue:
type: h2
repository:
type: h2
datasources:
h2:
url: jdbc:h2:mem:public;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
username: sa
password: ""
driverClassName: org.h2.Driver
Connection pool size
The total number of connections opened to the database will depend on your chosen architecture. Each Kestra instance will open a pool of up to the maximum-pool-size
(10 by default), with a minimum size of the minimum-idle
(also set to 10 by default).
- If you deploy Kestra as a standalone server, it will open 10 connections to the database.
- If you deploy each Kestra component separately, it will open 40 connections to the database (10 per component).
- If you deploy each Kestra component separately with 3 replicas, it will open 120 connections to the database.
Usually, the default connection pool sizing is enough, as HikariCP is optimized to use a low number of connections.
Datasource properties
Here are the datasource configuration properties. For more details, check the HikariCP configuration documentation:
Properties | Type | Description |
---|---|---|
url | String | The JDBC connection string. |
catalog | String | The default catalog name to be set on connections. |
schema | String | The default schema name to be set on connections. |
username | String | The default username used. |
password | String | The default password to use. |
transaction-isolation | String | The default transaction isolation level. |
pool-name | String | The name of the connection pool. |
connection-init-sql | String | The SQL string that will be executed on all new connections when they are created, before they are added to the pool. |
connection-test-query | String | The SQL query to be executed to test the validity of connections. |
connection-timeout | Long | The maximum number of milliseconds that a client will wait for a connection from the pool. |
idle-timeout | Long | The maximum amount of time (in milliseconds) that a connection is allowed to sit idle in the pool. |
minimum-idle | Long | The minimum number of idle connections that HikariCP tries to maintain in the pool, including both idle and in-use connections. Defaults to the value of maximum-pool-size |
initialization-fail-timeout | Long | The pool initialization failure timeout. |
leak-detection-threshold | Long | The amount of time that a connection can be out of the pool before a message is logged indicating a possible connection leak. |
maximum-pool-size | Int | The maximum size that the pool is allowed to reach, including both idle and in-use connections. Defaults to 10. |
max-lifetime | Long | The maximum lifetime of a connection in the pool. |
validation-timeout | Long | The maximum number of milliseconds that the pool will wait for a connection to be validated as alive. |
Here is the default HikariCP configuration:
transaction-isolation: default # driver default
pool-name: HikariPool-<Generated>
connection-init-sql: null
connection-test-query: null
connection-timeout: 30000 # 30 seconds
idle-timeout: 600000 # 10 minutes
minimum-idle: 10 # same as maximum-pool-size
initialization-fail-timeout: 1
leak-detection-threshold: 0
maximum-pool-size: 10
max-lifetime: 1800000 # 30 minutes
validation-timeout: 5000
Queues configuration
kestra.jdbc.queues
Kestra database queues simulate queuing doing long polling. They query a queues
table to detect new messages.
You can change these parameters to reduce the polling latency, but be aware it will increase the load on the database:
kestra.jdbc.queues.poll-size
: the maximum number of queues items fetched by each poll.kestra.jdbc.queues.min-poll-interval
: the minimum duration between 2 polls.kestra.jdbc.queues.max-poll-interval
: the maximum duration between 2 polls.kestra.jdbc.queues.poll-switch-interval
: the delay for switching from min-poll-interval to max-poll-interval when no message is received. (ex: when one message is received, themin-poll-interval
is used, if no new message arrived withinpoll-switch-interval
, we switch tomax-poll-interval
).
Here is the default configuration:
kestra:
jdbc:
queues:
poll-size: 100
min-poll-interval: 25ms
max-poll-interval: 1000ms
poll-switch-interval: 5s
kestra.jdbc.cleaner
Kestra cleans the queues
table periodically to optimize storage and performance. You can control how often you want this cleaning to happen, and how long messages should be kept in the queues
table .
Here is the default configuration:
kestra:
jdbc:
cleaner:
initial-delay: 1h
fixed-delay: 1h
retention: 7d
Protecting against too big messages
Note: this is an experimental feature available starting with Kestra 0.19.
The database backend has no limit on the size of messages it can handle. However, as messages are loaded into memory, this can endanger Kestra itself and push pressure on memory usage.
To prevent that, you can configure a functionality that will refuse to store too big messages and fail the execution instead.
The following configuration will refuse messages that exceed 1MiB by failing the execution.
kestra:
jdbc:
queues:
message-protection:
enabled: true
limit: 1048576
Was this page helpful?