Data Storage in Kestra
Understand where different data components (inputs, outputs, logs, and more) are stored in Kestra’s architecture.
Overview
Kestra processes and stores a variety of data, including flow definitions, workflow inputs, outputs, logs, execution metadata, and more. Understanding how these components are stored helps optimize performance, configure persistence, and integrate with external storage systems.
Kestra data is stored in either the repository, such as PostgreSQL, or in internal storage. By default, internal storage is local, but it can be configured to use services like AWS S3 or MinIO.
See Kestra architecture and internal storage for more details.
Data storage components
The table below outlines key data components, where they are stored, and their purpose.
| Data component | Storage location | Description |
|---|---|---|
| Flows & definitions | Repository | Stores flows, tasks, and their configurations. |
| Namespaces | Repository | Organizes workflows and manages secrets, plugin defaults, and variables. |
| Namespace files | Internal storage | Stores code and configuration files in Kestra’s storage backend. |
| Executions & metadata | Repository | Stores execution details including status, timestamps, and metadata. |
| Inputs | Internal storage | Stores files provided as inputs to a flow execution. |
| Input files | Internal storage | Stores additional files for script or CLI tasks. |
| Outputs | Internal storage | Stores outputs from tasks, separate from the database. |
| Output files | Internal storage | Stores generated files for download and reuse in downstream tasks. |
| Key-value pairs | Internal storage & repository (metadata only) | KV store holds data in key-value format. Metadata is recorded in the repository. |
| Logs & audit logs (Enterprise Edition) | Repository | Stores logs generated by tasks. |
| Task state & variables | Repository | Stores dynamic variables and task states during executions. |
| Secrets | Repository or external secret manager | Stores secrets internally or integrates with services like AWS Secrets Manager, Vault, or Google Secret Manager. |
| Queues | Repository or Kafka | Handles internal communication between Kestra components. |
| Triggers | Repository | Stores definitions of event-based triggers. |
| User administration | Repository | Stores RBAC, user management, and related metadata. |
Internal storage
Internal storage is used for handling files during executions. It ensures efficient input and output management without burdening the database.
- Purpose: Handles inputs, outputs, temporary execution data, and artifacts such as namespace files.
- KV store: Stores key-value pairs in internal storage, with metadata in the repository. Metadata includes the key, URI, TTL, and timestamps.
- Backends: By default, Kestra uses local storage, but for production you can configure cloud storage such as:
- AWS S3
- Google Cloud Storage
- Azure Blob Storage
- MinIO
- Any S3-compatible service
Configuring internal storage
Example docker-compose.yaml configuration for AWS S3:
kestra:
storage:
type: s3
bucket: "kestra-internal-storage"
region: "us-east-1"
For full details, see internal storage configuration.
Additional information
Flows & execution metadata
- Stored in PostgreSQL, MySQL, or H2 (not recommended for distributed components).
- Includes:
- Flow definitions
- Execution details
- Execution queues
- Historical metadata
- Accessible via the Kestra API and UI.
Logs
- Open source: Logs are stored in the database.
- Enterprise Edition: Supports Elasticsearch as a log backend, in addition to the database.
- Audit logs are stored in the repository.
- Logs can be accessed through the API, UI, or external logging integrations such as the log shipper.
Queues
- Open source: Stored in the database.
- Enterprise Edition: Can use Kafka for inter-component messaging.
Secrets management
Secrets can be stored in Kestra’s database or in external managers like AWS Secrets Manager, Google Secret Manager, Azure Key Vault, or HashiCorp Vault.
Example configuration for AWS Secret Manager:
kestra:
secret:
type: aws-secret-manager
aws-secret-manager:
access-key-id: my-access-key
secret-key-id: my-secret-key
sessionToken: my-session-token
region: us-east-1
See secret managers for more.
Database maintenance
Since databases accumulate large amounts of execution and log data, it’s recommended to use purge tasks to free up storage and maintain performance.
Conclusion
Kestra separates structured execution data from file-based artifacts to keep performance optimal:
- Database: Stores flows, metadata, logs, and secrets.
- Internal storage: Stores inputs, outputs, and files.
For large-scale deployments, cloud storage backends ensure scalability.
To further separate data across business units or environments, see the governance features in the Enterprise Edition, including tenants.
Was this page helpful?