Configure Repository, Queue, Datasource & Internal Storage

Use this page when configuring the core runtime services that make Kestra run.

Core setup decisions

Every Kestra deployment must define:

• repository type
• queue type
• internal storage type

The common production path is PostgreSQL for queue and repository, plus an object store or durable internal storage backend.

Queues and repositories must stay compatible:

• in-memory queue with in-memory repository for local testing only
• JDBC queue with H2, MySQL, or PostgreSQL repository
• Kafka queue with Elasticsearch repository in Enterprise Edition

Database and datasources

Start here if you are choosing the persistence layer for a new Kestra instance or moving from a local setup to a durable environment. In most teams, this is the first configuration page they revisit after initial installation.

Use kestra.queue.type and kestra.repository.type to select your backend:

kestra:
  queue:
    type: postgres
  repository:
    type: postgres

Then define the datasource:

datasources:
  postgres:
    url: jdbc:postgresql://localhost:5432/kestra
    driver-class-name: org.postgresql.Driver
    username: kestra
    password: k3str4

The examples below are intentionally minimal. Use them to confirm the backend choice and basic connection shape first, then add pooling and operational settings afterward.

Minimal datasource examples:

kestra:
  queue:
    type: postgres
  repository:
    type: postgres

datasources:
  postgres:
    url: jdbc:postgresql://localhost:5432/kestra
    driver-class-name: org.postgresql.Driver
    username: kestra
    password: k3str4

kestra:
  queue:
    type: mysql
  repository:
    type: mysql

datasources:
  mysql:
    url: jdbc:mysql://localhost:3306/kestra
    driver-class-name: com.mysql.cj.jdbc.Driver
    username: kestra
    password: k3str4
    dialect: MYSQL

kestra:
  queue:
    type: h2
  repository:
    type: h2

datasources:
  h2:
    url: jdbc:h2:mem:public;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
    username: sa
    password: ""
    driver-class-name: org.h2.Driver

Use H2 for local development. For production, prefer PostgreSQL, or MySQL if PostgreSQL is not an option.

Connection pooling and JDBC queue tuning

Most users can keep the defaults here until they see either connection pressure or queue latency. This section matters most for larger deployments, split-component topologies, or databases that are already under load.

Kestra uses HikariCP for datasource pooling. Common options include:

Example:

datasources:
  postgres:
    url: jdbc:postgresql://localhost:5432/kestra
    driver-class-name: org.postgresql.Driver
    username: kestra
    password: k3str4
    maximum-pool-size: 20
    minimum-idle: 10

Rough connection planning:

• standalone server: about 10 connections
• split components: about 40 connections
• split components with 3 replicas: about 120 connections

JDBC queues long-poll the queues table. Lower intervals reduce latency but increase database load:

kestra:
  jdbc:
    queues:
      poll-size: 100
      min-poll-interval: 25ms
      max-poll-interval: 1000ms
      poll-switch-interval: 5s

The JDBC cleaner removes old queue rows:

kestra:
  jdbc:
    cleaner:
      initial-delay: 1h
      fixed-delay: 1h
      retention: 7d

To reject oversized JDBC messages before they create memory pressure:

kestra:
  jdbc:
    queues:
      message-protection:
        enabled: true
        limit: 1048576

If you are not troubleshooting queue throughput or database pressure, you can usually leave the JDBC queue settings alone and return to them only when scaling.

Internal storage

kestra.storage.type controls where Kestra stores internal files such as task outputs, namespace files, and execution artifacts. Choose the backend based on durability and whether all Kestra components can reach the same storage.

Supported backends: Local · AWS S3 · Google Cloud Storage · Azure Blob Storage · MinIO / S3-compatible · SeaweedFS · Cloudflare R2 · Huawei OBS

Storage isolation

Storage isolation restricts which Kestra services are permitted to access internal storage files directly:

kestra:
  storage:
    type: gcs
    isolation:
      enabled: true
      denied-services:
        - EXECUTOR

Local

kestra:
  storage:
    type: local
    local:
      base-path: /app/storage

Local storage works for standalone deployments with a persistent volume. In distributed deployments, it only works safely when all components share the same filesystem through a ReadWriteMany volume or equivalent shared storage. If that is not the case, use object storage instead.

AWS S3

Minimum configuration

kestra:
  storage:
    type: s3
    s3:
      bucket: my-kestra-bucket
      region: us-east-1

With explicit credentials:

kestra:
  storage:
    type: s3
    s3:
      bucket: my-kestra-bucket
      region: us-east-1
      access-key: YOUR_ACCESS_KEY
      secret-key: YOUR_SECRET_KEY

Configuration reference

Credential resolution order

1. access-key / secret-key in config.
2. sts-role-arn — role assumption chained on top of any resolved identity.
3. AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY environment variables.
4. AWS CLI profile (~/.aws/credentials).
5. EKS Pod Identity or IRSA (IAM Roles for Service Accounts).
6. EC2/ECS instance metadata profile.

Permissions

Grant the IAM identity Kestra uses (user, role, or instance profile) the following permissions on the bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::YOUR_BUCKET"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::YOUR_BUCKET/*"
    }
  ]
}

If s3-files-compatible: true is set, also include s3:PutBucketVersioning in the bucket-level statement. If sts-role-arn is set, the calling identity additionally needs sts:AssumeRole on that role ARN.

Google Cloud Storage

Minimum configuration

kestra:
  storage:
    type: gcs
    gcs:
      bucket: my-kestra-bucket
      project-id: my-gcp-project

With an explicit service account key:

kestra:
  storage:
    type: gcs
    gcs:
      bucket: my-kestra-bucket
      project-id: my-gcp-project
      service-account: |
        { "type": "service_account", ... }

Configuration reference

Credential resolution order

1. service-account JSON key in config.
2. GOOGLE_APPLICATION_CREDENTIALS environment variable pointing to a key file.
3. Google default application credentials (Workload Identity on GKE, GCE metadata server, gcloud CLI).

Permissions

Assign the roles/storage.objectAdmin predefined role on the bucket (not the project) to the service account or workload identity principal. Fine-grained equivalent:

Azure Blob Storage

Minimum configuration

kestra:
  storage:
    type: azure
    azure:
      endpoint: https://myaccount.blob.core.windows.net
      container: kestra-storage

Configuration reference

Connection

Connection string auth

Shared key auth

SAS token auth

Managed identity auth

Workload identity auth

All auth modes

Credential resolution order

1. connection-string — if set, all other auth properties are skipped.
2. shared-key-account-name + shared-key-account-access-key.
3. sas-token.
4. DefaultAzureCredential chain — environment variables (AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID), then managed identity (honoring managed-identity-client-id / managed-identity-resource-id), then workload identity (honoring workload-identity-client-id), then Azure CLI.

Permissions

Assign the Storage Blob Data Contributor built-in role to the identity (service principal, managed identity, or workload identity principal) on the container or storage account. This role covers all data-plane read, write, and delete operations on blobs.

Connection string and shared key auth carry full data-plane access inherently and do not require a role assignment.

MinIO / S3-compatible

Use type: minio for MinIO and any S3-compatible object storage — including Ceph, SeaweedFS S3 API, Garage, and Outscale OOS.

Minimum configuration

kestra:
  storage:
    type: minio
    minio:
      endpoint: my.minio.domain.com
      port: 9000
      bucket: kestra-storage
      access-key: YOUR_ACCESS_KEY
      secret-key: YOUR_SECRET_KEY

Configuration reference

If MinIO uses MINIO_DOMAIN, set vhost: true and point endpoint at the base domain rather than bucket.domain.

Permissions

Create a MinIO policy that grants the access key read/write access to the bucket, then attach it via mc admin policy attach:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::YOUR_BUCKET"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::YOUR_BUCKET/*"
    }
  ]
}

For Ceph RGW, SeaweedFS, or other S3-compatible backends, apply the equivalent bucket ACL or user policy through that system’s admin interface.

SeaweedFS

kestra:
  storage:
    type: seaweedfs
    seaweedfs:
      filer-host: localhost
      filer-port: 18888

SeaweedFS has no built-in authentication. Control access at the network level by restricting which hosts can reach the filer gRPC port.

Cloudflare R2

kestra:
  storage:
    type: cloudflare
    cloudflare:
      account-id: YOUR_CLOUDFLARE_ACCOUNT_ID
      bucket: kestra-storage
      access-key-id: YOUR_R2_ACCESS_KEY_ID
      secret-access-key: YOUR_R2_SECRET_ACCESS_KEY

Permissions

Create an R2 API token with Object Read & Write permissions. Scope the token to the specific bucket rather than all buckets. R2 does not use IAM-style policies — permissions are set directly on the API token in the Cloudflare dashboard under R2 → Manage R2 API Tokens.

Huawei OBS

Kestra supports Huawei Cloud Object Storage Service (OBS) as an internal storage backend.

Minimum configuration

kestra:
  storage:
    type: obs
    obs:
      access-key: YOUR_OBS_ACCESS_KEY
      secret-key: YOUR_OBS_SECRET_KEY
      bucket: kestra-storage
      region: cn-north-4

Configuration reference

Permissions

Attach a bucket policy or Huawei IAM policy granting the AK/SK identity the following actions on the bucket and its objects:

Alternatively, assign the predefined OBSFullAccess Huawei IAM policy scoped to the bucket if a custom bucket policy is not required.

Server, environment, and JVM settings

These settings shape how the instance presents itself and how the Java process behaves at runtime. They are less about feature enablement and more about making the deployment fit its environment.

Common runtime areas include:

• kestra.server.* for basic auth and liveness
• kestra.url for the instance URL
• kestra.environment.* for environment display metadata
• JAVA_OPTS for JVM tuning such as timezone and heap settings
• kestra.variables.* for global variables and recursive rendering behavior

Environment metadata shown in the UI:

kestra:
  environment:
    name: Production
    color: "#FCB37C"

JVM settings are usually passed through JAVA_OPTS:

export JAVA_OPTS="-Duser.timezone=Europe/Paris -Xmx1g"

Common uses include:

• setting user.timezone to control scheduling and log display
• setting a fixed heap with -Xmx
• configuring Java proxy settings for outbound access

Global variables and rendering behavior also live here:

kestra:
  variables:
    env-vars-prefix: ENV_
    globals:
      region: eu-west-1
    recursive-rendering: true
    cache-enabled: true

env-vars-prefix controls which environment variables become available in expressions under envs.*. For example, ENV_MY_VARIABLE becomes {{ envs.my_variable }}.

Use globals for values that need to be available in every flow, recursive-rendering only when you intentionally want pre-0.14 recursive behavior, and cache-enabled when you need to trade CPU for correctness while debugging template changes. Set cache-size to limit the number of cached templates (default 1000):

kestra:
  variables:
    cache-size: 1000

Optional runtime features

These settings are not part of the core queue or repository setup, but they do matter in real deployments.

Some notifications and generated links depend on kestra.url being set to the public base URL without /ui or /api:

kestra:
  url: https://www.my-host.com/kestra/

The web UI can also be customized at runtime:

kestra:
  webserver:
    google-analytics: UA-12345678-1
    html-head: |
      <style type="text/css">
        .v-sidebar-menu .logo:after {
          content: "Local";
        }
      </style>

Use html-head sparingly for environment banners, extra CSS, or internal scripts that must load with the app shell.

To allow universal file access from host-mounted paths, both mount the directory and add it to the allowlist:

kestra:
  local-files:
    allowed-paths:
      - /scripts
    enable-preview: false

Without the allowlist, file-access URIs pointing at local host paths will be rejected even if the path is mounted into the container.

When to use this page

• Need logs, telemetry, metrics, endpoints, CORS, or SSL: Observability and Networking
• Need plugin defaults, retries, task temp storage, templates, or system flows: Plugins and Execution
• Need secret backends or server hardening: Security and Secrets