How Kestra Stores Inputs, Outputs, Logs, and Metadata

Understand where different data components (inputs, outputs, logs, and more) are stored in Kestra’s architecture.

Kestra processes and stores a variety of data, including flow definitions, workflow inputs, outputs, logs, execution metadata, and more. Understanding how these components are stored helps optimize performance, configure persistence, and integrate with external storage systems.

Kestra data is stored in either the repository, such as PostgreSQL, or in internal storage. By default, internal storage is local, but you can configure it to use services like AWS S3 or MinIO.

Data storage components

The table below outlines key data components, where they are stored, and their purpose.

Internal storage

Internal storage is used for handling files during executions. It ensures efficient input and output management without burdening the database.

• Purpose: Handles inputs, outputs, temporary execution data, and artifacts such as namespace files.
• KV store: Stores key-value pairs in internal storage, with metadata in the repository. Metadata includes the key, URI, TTL, and timestamps.
• Backends: By default, Kestra uses local storage, but for production you can configure cloud storage such as:
  • AWS S3
  • Google Cloud Storage
  • Azure Blob Storage
  • MinIO
  • Any S3-compatible service

Configuring internal storage

Example docker-compose.yaml configuration for AWS S3:

kestra:
  storage:
    type: s3
    bucket: "kestra-internal-storage"
    region: "us-east-1"

For full details, see internal storage configuration.

Additional information

Flows and execution metadata

• Stored in PostgreSQL, MySQL, or H2 (not recommended for distributed components).
• Includes:
  • Flow definitions
  • Execution details
  • Execution queues
  • Historical metadata
• Accessible via the Kestra API and UI.

Logs

• Open source: Logs are stored in the database.
• Enterprise Edition: Supports Elasticsearch as a log backend, in addition to the database.
  • Audit logs are stored in the repository.
• Logs can be accessed through the API, UI, or external logging integrations such as the log shipper.

Queues

• Open source: Stored in the database.
• Enterprise Edition: Can use Kafka for inter-component messaging.

Secrets management

Secrets can be stored in Kestra’s database or in external managers like AWS Secrets Manager, Google Secret Manager, Azure Key Vault, or HashiCorp Vault.

Example configuration for AWS Secret Manager:

kestra:
  secret:
    type: aws-secret-manager
    aws-secret-manager:
      access-key-id: my-access-key
      secret-key-id: my-secret-key
      sessionToken: my-session-token
      region: us-east-1

See secret managers for more.

Handling sensitive data and PII

Understanding where data is persisted is critical when flows process personally identifiable information (PII) or other sensitive values.

Stored as plaintext in the database:

• Non-sensitive scalar inputs (STRING, INTEGER, etc.) — stored in the executions table
• Task output values emitted via outputs in the script output protocol — stored in the task_outputs table
• Log messages — stored in the logs table

Stored encrypted in the database:

• SECRET type inputs — stored in the executions table as an encrypted value (requires encryption to be configured); the value is also automatically masked in logs
• encryptedOutputs values (Enterprise Edition) — stored in the task_outputs table

Not stored in the database:

• FILE-type inputs and output files — these go to internal storage (your configured S3, GCS, Azure Blob, etc.).

Encrypting sensitive task outputs

In Enterprise Edition, script tasks support an encryptedOutputs key in the ::{}:: output protocol. Values written this way are wrapped in an EncryptedString and stored encrypted in the task_outputs table rather than as plaintext. They are merged into the same outputs map as regular outputs and are decrypted by Kestra at evaluation time.

id: sensitive_data_flow
namespace: company.team

inputs:
  - id: ssn
    type: SECRET

tasks:
  - id: process_pii
    type: io.kestra.plugin.scripts.shell.Script
    script: |
      echo '::{"encryptedOutputs":{"ssn":"{{ inputs.ssn }}"}}::'
      echo '::{"outputs":{"status":"processed"}}::'

Both plaintext and encrypted outputs are accessible in downstream tasks via {{ outputs.process_pii.ssn }} — Kestra decrypts the value at expression evaluation time.

Lines matching the ::{}:: protocol are consumed by the output processor and never written to the logs table, so the plaintext value is not exposed in log storage.

Best practices for PII-sensitive flows

• Use type: SECRET for sensitive inputs. SECRET inputs are encrypted at rest in the executions table and automatically masked in log output. Requires kestra.encryption.secret-key to be configured — without it, SECRET inputs fail at runtime.
• Use encryptedOutputs for sensitive task outputs (Enterprise Edition). Values are stored encrypted in the task_outputs table and decrypted transparently when referenced in downstream tasks.
• Use Secrets for credentials and configuration values that should never appear in execution records.
• Log carefully. Only SECRET inputs are automatically masked in logs. Any sensitive value logged directly — via print, echo, or a logging statement — is stored as plaintext in the logs table.

Database maintenance

Use purge tasks to free up storage and maintain performance as databases accumulate execution and log data.

To further separate data across business units or environments, see the governance features in the Enterprise Edition, including tenants.