Store Execution Logs Outside Your Main Database

The external log data store routes Kestra execution logs to a dedicated backend, separate from the main database. Supported backends are JDBC (PostgreSQL, MySQL, H2), Elasticsearch, Splunk, and Datadog.

By default, Kestra stores execution logs in the same database as flows, executions, and state.

Why move logs out of the main database

Logs are high-volume, write-heavy data. Keeping them in the main database creates three operational problems:

• Database bloat: Log volume grows independently of flows and executions, inflating main database size over time.
• Migration risk: Kestra schema migrations run across every table, including logs. Large log tables extend upgrade downtime.
• Retention mismatch: Flows, executions, and logs typically have different retention requirements but share the same database and backup lifecycle.

How it works

The Indexer writes logs to the configured log store in real time. The log UI and dashboards read from the same store. The main database stores only flows, executions, and state.

Default behavior

If kestra.logs.type is not set, logs stay in the main backend (kestra.repository.type). Existing installations see no change on upgrade.

The full resolution order is:

1. kestra.logs.type — explicit log store selection
2. kestra.repository.type — fallback: logs stay in the main backend (the default)
3. If neither is configured, Kestra fails fast at startup with a clear error

Migration note

Configuring an external log store applies to new executions only. Historical logs remain in the main database.

Configure the JDBC log store

Select a JDBC store by setting kestra.logs.type to h2, postgres, or mysql. The store can connect to a dedicated log database or reuse the main datasource.

Config reference

When kestra.logs.<type>.url is not set, the JDBC store reuses the primary datasource and kestra.logs.type must match kestra.repository.type. This is useful for verifying config shape without standing up a second database, but it does not move logs out of the main database.

Example: PostgreSQL backend with a separate PostgreSQL log database

Two independent Postgres databases: one for the main backend, one for logs only.

services:
  postgres:                         # main backend DB
    image: postgres:16
    environment:
      POSTGRES_DB: kestra
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: k3str4
    volumes:
      - postgres-data:/var/lib/postgresql/data

  postgres-logs:                    # dedicated log DB
    image: postgres:16
    environment:
      POSTGRES_DB: kestra_logs
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: k3str4
    volumes:
      - postgres-logs-data:/var/lib/postgresql/data

  kestra:
    image: kestra/kestra:latest
    command: server standalone
    depends_on: [postgres, postgres-logs]
    ports: ["8080:8080"]
    environment:
      KESTRA_CONFIGURATION: |
        datasources:
          postgres:
            url: jdbc:postgresql://postgres:5432/kestra
            driverClassName: org.postgresql.Driver
            username: kestra
            password: k3str4
        kestra:
          repository:
            type: postgres
          queue:
            type: postgres
          storage:
            type: local
          logs:
            type: postgres
            postgres:
              url: jdbc:postgresql://postgres-logs:5432/kestra_logs
              username: kestra
              password: k3str4
              table: logs

volumes:
  postgres-data: {}
  postgres-logs-data: {}

The main backend keeps using datasources.postgres. The log store opens its own connection pool against postgres-logs, runs log-table migrations there, and routes all log reads and writes to that database.

Configure the Elasticsearch log store

The Elasticsearch log store reuses the existing kestra.elasticsearch.* connection config; you add a kestra.logs block naming the index.

Config reference

Example: PostgreSQL backend with Elasticsearch log store

The main backend stays Postgres; execution logs go to Elasticsearch.

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_DB: kestra
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: k3str4
    volumes:
      - postgres-data:/var/lib/postgresql/data

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
    environment:
      discovery.type: single-node
      xpack.security.enabled: "false"
      ES_JAVA_OPTS: "-Xms512m -Xmx512m"
    ports: ["9200:9200"]

  kestra:
    image: kestra/kestra:latest-ee           # EE image required
    command: server standalone
    depends_on: [postgres, elasticsearch]
    ports: ["8080:8080"]
    environment:
      KESTRA_CONFIGURATION: |
        datasources:
          postgres:
            url: jdbc:postgresql://postgres:5432/kestra
            driverClassName: org.postgresql.Driver
            username: kestra
            password: k3str4
        kestra:
          repository:
            type: postgres
          queue:
            type: postgres
          storage:
            type: local
          logs:
            type: elasticsearch
            elasticsearch:
              index: logs
          elasticsearch:
            client:
              http-hosts:
                - http://elasticsearch:9200
              # basic-auth:
              #   username: elastic
              #   password: changeme
            indices:
              logs:
                cls: io.kestra.core.models.executions.LogEntry
                index: logs

volumes:
  postgres-data: {}

Configure the Splunk log store

The Splunk log store sends execution logs to a Splunk instance via HTTP Event Collector (HEC) for writes and the Splunk management API for reads, aggregation, and purge.

Config reference

host and hecToken are required. For the management API, provide either token (bearer) or both username and password (basic auth).

Example: PostgreSQL backend with Splunk log store

kestra:
  repository:
    type: postgres
  queue:
    type: postgres
  logs:
    type: splunk
    splunk:
      host: splunk.example.com
      hecToken: "your-hec-token"
      token: "your-management-api-bearer-token"
      index: kestra

Configure the Datadog log store

The Datadog log store sends execution logs to the Datadog Log Intake API for writes and uses the Logs Search and Aggregate APIs for reads.

Config reference

apiKey and applicationKey are required.

Example: PostgreSQL backend with Datadog log store

kestra:
  repository:
    type: postgres
  queue:
    type: postgres
  logs:
    type: datadog
    datadog:
      apiKey: "your-datadog-api-key"
      applicationKey: "your-datadog-application-key"
      site: datadoghq.com

Store capabilities

Each store declares its capabilities, and Kestra adapts gracefully:

When a store does not support aggregation, dashboard log charts display “No data” rather than erroring. When a store does not support purge, Kestra’s PurgeLogs operations no-op for logs; the backend’s own retention or TTL policy governs deletion.

Log Shipper vs External Log Data Store

Use the External Log Data Store when you want logs out of the main database. Use the Log Shipper when you want logs in a third-party observability platform as well, regardless of where Kestra stores them internally.