docsarchitecture

Main components​Main components

Technical description of Kestra's main components, including the internal storage, queue, repository, and plugins.

Kestra has the following internal components:

  • The Internal Storage stores flow data like task outputs and flow inputs.
  • The Queue is used for internal communication between Kestra server components.
  • The Repository is used to store flows, templates, executions, logs, etc. The repository stores every internal object.
  • The Plugins extend the core of Kestra with new task and trigger types, storage implementations, data transformations, etc.

These internal components are provided on multiple implementations depending on your needs and deployment architecture. You may need to install additional plugins to use some implementations.

Internal Storage

The Internal Storage is a dedicated storage area used by Kestra to handle arbitrary-sized files during flow executions. It stores data both for inputs and outputs, enabling scalable file handling between tasks.

Purpose

Internal Storage is used to:

  • Store files generated during a flow's execution and pass data between tasks (outputs).
  • Automatically save flow inputs of type FILE.
  • Provide download links for stored files in the Outputs tab of an execution.

Files in Internal Storage can be retrieved from the execution context using {{ outputs.task_id.output_attribute }} (commonly the uri property). Kestra automatically fetches the file when needed.

Execution metadata — including file paths in Internal Storage — is recorded in the Repository.

Storage Types

By default, Kestra uses a local storage implementation, storing files in a directory on the host filesystem. This option is simple but not scalable and is not recommended for production.

Additional, scalable storage backends are available as plugins:

For more details, see the Internal Storage configuration documentation.

Queue

The Queue, or more precisely, queues, are used internally for communication between the different Kestra server components. Kestra provides multiple queue types that must be used with their repository counterparts.

There are three types of queues:

  • In-Memory: must be used with the In-Memory Repository
  • Database: must be used with the Database Repository
  • Kafka: must be used with the Elasticsearch Repository. Only available in the Enterprise Edition

Repository

The Repository, or more precisely, repositories, are the internal way to store data. Kestra provides multiple repository types that must be used with their queue counterparts.

There exist three types of repositories:

  • In-Memory: must be used with the In-Memory Queue
  • Database: must be used with the Database Queue
  • Elasticsearch: must be used with the Kafka Queue. Only available in the Enterprise Edition

Plugins

Kestra's core is not able to handle a lot of task types on its own. Therefore, we have designed a Plugins' ecosystem that allows developing as many task types as you need. A wide range of plugins are already available, and many more will be delivered by the Kestra team!

Plugins are also used to provide different implementations for Kestra's internal components like its Internal Storage.

Was this page helpful?