​Deployment architecture

Examples of deployment architectures, depending on your needs.

Kestra is a Java application distributed as an executable. It supports multiple deployment options:

• Docker
• Kubernetes
• Manual deployment

Kestra’s plugin system allows you to choose the dependency types that best match your requirements.
Below are three common deployment architectures.

Small-sized deployment

For small-scale deployments, you can use the Kestra standalone server, which runs all server components in a single process. This architecture has no scaling capability.

In this setup, a database is the only dependency, minimizing the stack to maintain. Supported databases include:

• PostgreSQL
• MySQL
• H2

Medium-sized deployment

For medium-scale deployments, where high availability is not required, Kestra can be run with a relational database (Postgres or MySQL) as the only dependency. H2 is not recommended in distributed setups.

• Supported databases: PostgreSQL and MySQL
• All server components communicate through the database

In this mode, if components are distributed across multiple hosts, you must use a shared internal storage implementation such as Google Cloud Storage, AWS S3, or Azure Blob Storage.

High-availability deployment

For high throughput and full horizontal and vertical scaling, the database is replaced with Kafka and Elasticsearch. This architecture removes single points of failure and enables scaling of all server components.

• Dependencies: Kafka and Elasticsearch
• Available only in the Enterprise Edition

As with medium deployments, a distributed internal storage solution is required if components run on different hosts.

Kafka

Kafka is the backbone of high availability mode, powering communication and scalability.

Kafka executor

The executor runs as a Kafka Streams application. It:

• Processes all events from Kafka in order
• Maintains the internal state of executions
• Merges task run results from workers
• Detects failed workers and resubmits their tasks

Executors scale horizontally within the limits of Kafka partitions. Since executors perform lightweight operations, they typically require minimal resources unless handling very high execution volumes.

Kafka worker

The worker runs as a Kafka consumer. It:

• Processes tasks assigned by executors
• Runs tasks in an internal thread pool
• Scales horizontally, with multiple instances across servers

If a worker fails, the executor detects it and resubmits the tasks to another available worker.

Elasticsearch

Elasticsearch acts as the database for Kestra’s webserver, providing fast search, aggregation, and retrieval of flows, executions, and logs.
It is only required in high availability mode and is used exclusively by the API and UI.