Data Storage in Kestra​Data ​Storage in ​Kestra

Understand where different data components (inputs, outputs, logs, and more) are stored in Kestra’s architecture.

Overview

Kestra processes and stores a variety of data, including flow definitions, workflow inputs, outputs, logs, execution metadata, and more. Understanding how these components are stored helps optimize performance, configure persistence, and integrate with external storage systems.

Kestra data is stored in either the repository, such as PostgreSQL, or in internal storage. By default, internal storage is local, but it can be configured to use services like AWS S3 or MinIO.

Data storage components

The table below outlines key data components, where they are stored, and their purpose.

Data componentStorage locationDescription
Flows & definitionsRepositoryStores flows, tasks, and their configurations.
NamespacesRepositoryOrganizes workflows and manages secrets, plugin defaults, and variables.
Namespace filesInternal storageStores code and configuration files in Kestra’s storage backend.
Executions & metadataRepositoryStores execution details including status, timestamps, and metadata.
InputsInternal storageStores files provided as inputs to a flow execution.
Input filesInternal storageStores additional files for script or CLI tasks.
OutputsInternal storageStores outputs from tasks, separate from the database.
Output filesInternal storageStores generated files for download and reuse in downstream tasks.
Key-value pairsInternal storage & repository (metadata only)KV store holds data in key-value format. Metadata is recorded in the repository.
Logs & audit logs (Enterprise Edition)RepositoryStores logs generated by tasks.
Task state & variablesRepositoryStores dynamic variables and task states during executions.
SecretsRepository or external secret managerStores secrets internally or integrates with services like AWS Secrets Manager, Vault, or Google Secret Manager.
QueuesRepository or KafkaHandles internal communication between Kestra components.
TriggersRepositoryStores definitions of event-based triggers.
User administrationRepositoryStores RBAC, user management, and related metadata.

Internal storage

Internal storage is used for handling files during executions. It ensures efficient input and output management without burdening the database.

  • Purpose: Handles inputs, outputs, temporary execution data, and artifacts such as namespace files.
  • KV store: Stores key-value pairs in internal storage, with metadata in the repository. Metadata includes the key, URI, TTL, and timestamps.
  • Backends: By default, Kestra uses local storage, but for production you can configure cloud storage such as:

Configuring internal storage

Example docker-compose.yaml configuration for AWS S3:

yaml
kestra:
  storage:
    type: s3
    bucket: "kestra-internal-storage"
    region: "us-east-1"

For full details, see internal storage configuration.

Additional information

Flows & execution metadata

  • Stored in PostgreSQL, MySQL, or H2 (not recommended for distributed components).
  • Includes:
    • Flow definitions
    • Execution details
    • Execution queues
    • Historical metadata
  • Accessible via the Kestra API and UI.

Logs

  • Open source: Logs are stored in the database.
  • Enterprise Edition: Supports Elasticsearch as a log backend, in addition to the database.
    • Audit logs are stored in the repository.
  • Logs can be accessed through the API, UI, or external logging integrations such as the log shipper.

Queues

  • Open source: Stored in the database.
  • Enterprise Edition: Can use Kafka for inter-component messaging.

Secrets management

Secrets can be stored in Kestra’s database or in external managers like AWS Secrets Manager, Google Secret Manager, Azure Key Vault, or HashiCorp Vault.

Example configuration for AWS Secret Manager:

yaml
kestra:
  secret:
    type: aws-secret-manager
    aws-secret-manager:
      access-key-id: my-access-key
      secret-key-id: my-secret-key
      sessionToken: my-session-token
      region: us-east-1

See secret managers for more.

Database maintenance

Since databases accumulate large amounts of execution and log data, it’s recommended to use purge tasks to free up storage and maintain performance.

Conclusion

Kestra separates structured execution data from file-based artifacts to keep performance optimal:

  • Database: Stores flows, metadata, logs, and secrets.
  • Internal storage: Stores inputs, outputs, and files.

For large-scale deployments, cloud storage backends ensure scalability.
To further separate data across business units or environments, see the governance features in the Enterprise Edition, including tenants.

Was this page helpful?