🚀 New! Kestra raises $3 million to grow Learn more

First, you need to configure Kestra to use a database for its Queue and Repository, for example for PostgreSQL:

yaml
kestra:
  queue:
    type: postgres
  repository:
    type: postgres

Currently, Kestra supports three databases: H2, MySQL, and PostgreSQL. H2 can be convenient for local development, but MySQL or PostgreSQL must be used in production.


Minimal configuration

The most important thing is to configure the way Kestra connects to a database, what is called a datasource. Under the cover, Kestra uses The HikariCP connection pool; each configuration options described inside the HikariCP documentation can be configured.


PostgreSQL

Here is a minimal configuration for PostgreSQL:

yaml
kestra:
  queue:
    type: postgres
  repository:
    type: postgres

datasources:
  postgres:
    url: jdbc:postgresql://localhost:5432/kestra
    driverClassName: org.postgresql.Driver
    username: kestra
    password: k3str4

MySQL

Here is a minimal configuration for MySQL:

yaml
kestra:
  queue:
    type: mysql
  repository:
    type: mysql

datasources:
  mysql:
    url: jdbc:mysql://localhost:3306/kestra
    driverClassName: com.mysql.cj.jdbc.Driver
    username: kestra
    password: k3str4
    dialect: MYSQL

H2

Here is a minimal configuration for H2:

yaml
kestra:
  queue:
    type: h2
  repository:
    type: h2

datasources:
  h2:
    url: jdbc:h2:mem:public;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
    username: sa
    password: ""
    driverClassName: org.h2.Driver

Connection pool size

The total number of connections opened to the database will depend on the deployment architecture used. Each Kestra instance will open a pool of up to the maximum-pool-size setting of 10 by default, with a minimum size of the minimum-idle setting of 10 by default.

For example:

  • If you deploy Kestra as a standalone server, it will open 10 connections to the database.
  • If you deploy each Kestra component separately, it will open 40 connections to the database (10 per component).
  • If you deploy each Kestra component separately with 3 replicas, it will open 120 connections to the database.

Usually, the default connection pool sizing is enough, as HikariCP is optimized to use a low number of connections.


Datasource properties

Here are the datasource configuration properties, for more information read the HikariCP configuration documentation:

PropertiesTypeDescription
urlStringThe JDBC connection string.
catalogStringThe default catalog name to be set on connections.
schemaStringThe default schema name to be set on connections.
usernameStringThe default username used.
passwordStringThe default password to use.
transaction-isolationStringThe default transaction isolation level.
pool-nameStringThe name of the connection pool.
connection-init-sqlStringThe SQL string that will be executed on all new connections when they are created, before they are added to the pool.
connection-test-queryStringThe SQL query to be executed to test the validity of connections.
connection-timeoutLongThe maximum number of milliseconds that a client will wait for a connection from the pool.
idle-timeoutLongThe maximum amount of time (in milliseconds) that a connection is allowed to sit idle in the pool.
minimum-idleLongThe minimum number of idle connections that HikariCP tries to maintain in the pool, including both idle and in-use connections. Defaults to the value of maximum-pool-size
initialization-fail-timeoutLongThe pool initialization failure timeout.
leak-detection-thresholdLongThe amount of time that a connection can be out of the pool before a message is logged indicating a possible connection leak.
maximum-pool-sizeIntThe maximum size that the pool is allowed to reach, including both idle and in-use connections. Defaults to 10.
max-lifetimeLongThe maximum lifetime of a connection in the pool.
validation-timeoutLongThe maximum number of milliseconds that the pool will wait for a connection to be validated as alive.

Here is the default HikariCP configuration:

yaml
transaction-isolation: default # driver default
pool-name: HikariPool-<Generated>
connection-init-sql: null
connection-test-query: null
connection-timeout: 30000 # 30 seconds
idle-timeout: 600000 # 10 minutes
minimum-idle: 10 # same as maximum-pool-size
initialization-fail-timeout: 1
leak-detection-threshold: 0
maximum-pool-size: 10
max-lifetime: 1800000 # 30 minutes
validation-timeout: 5000

Queues configuration

kestra.jdbc.queues

Kestra database queues simulate queuing doing long polling. They query a queues table to detect new messages.

You can change these parameters to reduce the polling latency, but be aware it will increase the load on the database:

  • kestra.jdbc.queues.poll-size: the maximum number of queues items fetched by each poll.
  • kestra.jdbc.queues.min-poll-interval: the minimum duration between 2 polls.
  • kestra.jdbc.queues.max-poll-interval: the maximum duration between 2 polls.
  • kestra.jdbc.queues.poll-switch-interval: the delay for switching from min-poll-interval to max-poll-interval when no message is received. (ex: when one message is received, the min-poll-interval is used, if no new message arrived within poll-switch-interval, we switch to max-poll-interval).

Here is the default configuration:

yaml
kestra:
  jdbc:
    queues:
      poll-size: 100
      min-poll-interval: 25ms
      max-poll-interval: 1000ms
      poll-switch-interval: 5s

kestra.jdbc.cleaner

Kestra cleans the queues table periodically to avoid infinite growth. You can control how often you want this cleaning to happen, and how long messages should be kept in the queues table .

Here is the default configuration:

yaml
kestra:
  jdbc:
    cleaner:
      initial-delay: 1h
      fixed-delay: 1h
      retention: 7d

Database migrations

Automatic database migration

Kestra uses Flyway to automatically perform database migrations when Kestra server is started. Flyway is a tool that allows version controlling changes to a database (such as Kestra's database) so that you can migrate it to a new version easily. Kestra stores the current version of the database in a table called flyway_schema_history. When Flyway runs, it compares the current version of the database with the version that it should be at. If the versions do not match, Flyway will run the necessary migrations to bring the database up to date. This process is automatic when Kestra server starts, therefore no manual intervention is required.

Manual database migration

Sometimes a manual database migration is useful, especially when you have a large database and you want to perform the migration before upgrading Kestra to avoid a long downtime.

For example, when migrating from Kestra v0.12.0 to v0.13.0, all indexes will be rebuilt due to addition of the multi-tenancy feature (the tenant_id column is added on almost every table). When using the JDBC runner with a large database, database migration can take multiple hours. In such use cases, we recommend performing the migration manually before starting Kestra by using the kestra sys database migrate command.

This command should use the same configuration as configured on your Kestra instance. Depending on whether you deploy Kestra using Docker or Kubernetes, this command can be launched via a docker exec or a kubectl exec command.

There are two ways to initiate the manual database migration:

  1. Keep Kestra running in an old version. Then, stop Kestra and launch the command on the new version.
  2. Start Kestra on the new version with automatic schema migration disabled via flyway.datasources.postgres.enabled=false (in case you're database is not postgres replace postgres with the type of your database) and launch the command: kestra sys database migrate.

Here is an example of how to launch the command via a docker exec command:

bash
docker exec your_container_id bash ./kestra sys database migrate --help

Here is the output of that --help command:

bash
Usage: kestra sys database migrate [-hVv] [--internal-log] [-c=<config>]
                                   [-l=<logLevel>] [-p=<pluginsPath>]
Force database schema migration.
Kestra uses Flyway to manage database schema evolution, this command will run
Flyway then exit.
  -c, --config=<config>   Path to a configuration file, default: /root/.
                            kestra/config.yml)
  -h, --help              Show this help message and exit.
      --internal-log      Change also log level for internal log, default:
                            false)
  -l, --log-level=<logLevel>
                          Change log level (values: TRACE, DEBUG, INFO, WARN,
                            ERROR; default: INFO)
  -p, --plugins=<pluginsPath>
                          Path to plugins directory , default: ./plugins)
  -v, --verbose           Change log level. Multiple -v options increase the
                            verbosity.
  -V, --version           Print version information and exit.