​Managing ​Upgrades

Kestra is a fast-evolving project. This section will guide you through the process of upgrading your Kestra installation.

How to upgrade Kestra

To upgrade Kestra, follow these steps:

1. Perform a database backup (optional but recommended)
2. Read the release notes on GitHub to understand the changes in the new version
3. Perform a rolling upgrade of Kestra components e.g. for a Kubernetes deployment, upgrade the Kestra Helm chart as described below in the section "Rolling upgrades in Kubernetes"
4. Perform any actions needed following the release notes (e.g. update configuration files, adjust deprecated features, etc.)

How to rollback Kestra to a previous version

Soemtimes you might need to downgrade Kestra to a previous version. Here are some steps to follow:

1. Perform a database backup (optional but recommended)
2. Stop all Kestra components
3. Restore from a backup
4. Restart from an older version

Check the Backup and Restore section for more information on how to backup and restore Kestra.

Where you can find the release changelog

You can find the release changelog on the main repository's Releases page. The changelog includes a full list of changes, new features, and bug fixes for each release, as well as any breaking changes that may require your attention. For a high-level eplanation of the changes, you can also check release blog posts.

How to identify breaking changes in a release

Next to all bug fixes and enhancements, you can find a dedicated section called Breaking Changes in the release notes. This section lists changes that may require some adjustments in your code or Kestra configuration, along with links to the documentation showing how to migrate.

How to minimise downtime when updating Kestra

If running Kestra in separate components you should:

• Stop the executors and the scheduler
• Stop the workers - there is a graceful shutdown (which can be configured but I'm not sure we already document how to configure this) and automatic task resubmission.
• Stop the webserver (and the indexer using EE and Kafka)

Normally, there is a graceful shutdown on all components so you will not lose anything. Once this is done, you can update and restart everything in the opposite order (or any order as all components are independent).

How to stick to a specific Kestra version

If you want to stick to a specific Kestra version, you can pin the Docker image tag to a specific release. Here are some examples:

• kestra/kestra:v0.14.4 includes the 0.14.4 release with the fourth patch version
• kestra/kestra:v0.14.4-full includes the 0.14.4 release with all plugins
• kestra/kestra:v0.15.0 includes the 0.15 release without any plugins
• kestra/kestra:v0.15.0-full includes the 0.15 release with all plugins.

Note that you can always create a custom image with your own plugins and package dependencies, as explained in the Docker installation.

Migrating a Standalone Installation

If you use a manual standalone installation with Java, you can download the Kestra binary for a specific version from the Assets menu of a specific Release page. The image below shows how you can download the binary for the 0.14.1 release.

Once you downloaded the binary, you can start kestra from that binary using the following command:

./kestra-VERSION server standalone

Migrating an installation with Docker

If you use Docker, you can change the Docker image tag to the desired version and restart the container(s) or Kubernetes pod(s).

Docker Compose

If you use Docker compose, adjust your Docker Compose file to use the desired Docker image tag and run docker-compose up -d to restart the container(s).

Migration in Kubernetes using Helm

If you use Helm, adjust the Helm chart tag value to point the installation to the desired version. For example, you can run the following command to upgrade the installation to the desired version:

helm upgrade kestra kestra/kestra --set image.tag=v0.15.0-full

For more complex configurations that include multiple changes, consider using a custom values file:

1. First, create a values.yaml file that contains the settings you want to adjust.
2. Then, use the helm upgrade command with the -f flag to specify your custom values file:

helm upgrade kestra kestra/kestra -f values.yaml

Rolling upgrades in Kubernetes

Upgrading Kestra on a Kubernetes cluster depends on a deployment rollout strategy.

Every service can be rolled out without any downtime except for a worker that needs a special attention.

Each standard component during the rollout will create a pod with a new version (keeping the old one running). When the new pod is up and running (passing all health checks), Kubernetes will shutdown the previous one leading to a zero-downtime migration.

Upgrading workers is more involved since workers handle data processing tasks which can range from a few seconds to many hours. You need to define the behavior for these tasks.

By default, Kestra worker process will wait until the completion of all task runs before shutting down during a migration. However, you can overwrite that default behavior if you wish. Kestra Helm charts provide a configuration of a terminationGracePeriodSeconds (set to 60 seconds by default) that allows you to define the amount of time you want to wait before force-killing the worker.

If the worker has no workers or is able to terminate all tasks before the grace period, it will be shutdown directly. If the pod is not able to finish the tasks affected before terminationGracePeriodSeconds, Kubernetes will kill the pod, leading to some tasks being resubmitted and handled by another worker.

In Kestra, every worker that died unexpectedly will be detected by the executor, and all unfinished task runs will be resubmitted and will be picked up by a new worker. In case of rollout with terminationGracePeriodSeconds, we are in the case of unexpected failure and the task will also be resubmitted.

Where can I find migration guides

The Migrations section includes detailed information about deprecated features and provides guidance on how to migrate from the deprecated to a new behavior.

For all breaking changes, the migration guides are linked in the release notes.

How to stay informed about new releases

You can get notified about new releases in the following ways:

1. Subscribe to notifications in the #announcements channel in the Slack community.
2. Follow us on Twitter
3. Follow us on LinkedIn
4. Subscribe to the Kestra newsletter
5. Subscribe to Release notifications on the main GitHub repository, as shown in the image below:

Database Migrations

There are two types of database migrations: automatic and manual.

Automatic database migration

Kestra uses Flyway to automatically perform database migrations when Kestra server is started. Flyway is a tool that allows version controlling changes to a database (such as Kestra's database) so that you can migrate it to a new version easily. Kestra stores the current version of the database in a table called flyway_schema_history. When Flyway runs, it compares the current version of the database with the version that it should be at. If the versions do not match, Flyway will run the necessary migrations to bring the database up to date. This process is automatic when Kestra server starts, therefore no manual intervention is required.

Manual database migration

Sometimes a manual database migration is useful, especially when you have a large database and you want to perform the migration before upgrading Kestra to avoid a long downtime.

For example, when migrating from Kestra v0.12.0 to v0.13.0, all indexes will be rebuilt due to addition of the multi-tenancy feature (the tenant_id column is added on almost every table). When using the JDBC runner with a large database, database migration can take multiple hours. In such use cases, we recommend performing the migration manually before starting Kestra by using the kestra sys database migrate command.

This command should use the same configuration as configured on your Kestra instance. Depending on whether you deploy Kestra using Docker or Kubernetes, this command can be launched via a docker exec or a kubectl exec command.

There are two ways to initiate the manual database migration:

1. Keep Kestra running in an old version. Then, stop Kestra and launch the command on the new version.
2. Start Kestra on the new version with automatic schema migration disabled via flyway.datasources.postgres.enabled=false (in case you're database is not postgres replace postgres with the type of your database) and launch the command: kestra sys database migrate.

Here is an example of how to launch the command via a docker exec command:

docker exec your_container_id bash ./kestra sys database migrate --help

Here is the output of that --help command:

Usage: kestra sys database migrate [-hVv] [--internal-log] [-c=<config>]
                                   [-l=<logLevel>] [-p=<pluginsPath>]
Force database schema migration.
Kestra uses Flyway to manage database schema evolution, this command will run
Flyway then exit.
  -c, --config=<config>   Path to a configuration file, default: /root/.
                            kestra/config.yml)
  -h, --help              Show this help message and exit.
      --internal-log      Change also log level for internal log, default:
                            false)
  -l, --log-level=<logLevel>
                          Change log level (values: TRACE, DEBUG, INFO, WARN,
                            ERROR; default: INFO)
  -p, --plugins=<pluginsPath>
                          Path to plugins directory , default: ./plugins)
  -v, --verbose           Change log level. Multiple -v options increase the
                            verbosity.
  -V, --version           Print version information and exit.

Getting help

If you have any questions about the upgrade process:

• if you are a Kestra Enterprise customer, please submit a Support Ticket
• reach out to us via Slack.

We understand that upgrades can be difficult. If you need more help, reach out to us and we'll help you with the migration based on your specific environment and use case.