Deploy Kestra with Docker Compose – Postgres Backend

Table of Contents

Start Kestra with a PostgreSQL database backend by using a Docker Compose file.

Deploy Kestra with Docker Compose and PostgreSQL

The quickest way to a production-ready, lightweight Kestra installation is to leverage Docker and Docker Compose. This guide helps you get started with Kestra using Docker.

Enterprise Edition images — log in to the private registry with your license credentials before pulling images:

docker login registry.kestra.io --username $LICENSEID --password $FINGERPRINT

Use registry.kestra.io/docker/kestra-ee:latest for the newest image, or pin a specific version such as registry.kestra.io/docker/kestra-ee:v1.0. See the Enterprise documentation and configuration requirements for deployment prerequisites. Compare editions in Open Source vs Enterprise if you are deciding between versions.

Before you begin

Make sure you have the following installed:

Download the Docker Compose file

Download the Docker Compose file using the following command on Linux and macOS:

curl -o docker-compose.yml \
https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml

On Windows, use the following command:

Invoke-WebRequest -Uri "https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml" -OutFile "docker-compose.yml"

You can also download the Docker Compose file manually and save it as docker-compose.yml.

Launch Kestra

Use the following command to start the Kestra server:

docker compose up -d

Open the URL http://localhost:8080 in your browser to launch the UI.

Adjusting the configuration

The command from the previous section starts a standalone server, with all architectural components running in one JVM.

The configuration is done inside the KESTRA_CONFIGURATION environment variable of the Kestra container. You can update the environment variable inside the Docker Compose file or pass it as a Docker CLI argument.

If you want to extend your Docker Compose file, modify container networking, or if you have any other issues using this Docker Compose file, check the Troubleshooting Guide.

For running Kestra in Docker Compose with each server component as a separate service, see the multi-component Docker Compose example.

Use a configuration file

If you want to use a configuration file instead of the KESTRA_CONFIGURATION environment variable to configure Kestra, you can update the default docker-compose.yml.

First, create a configuration file containing the KESTRA_CONFIGURATION environment variable defined in the docker-compose.yml file. You can name it application.yaml.

Next, update the kestra service in the docker-compose.yml file to mount this file into the container and start up Kestra using the --config option:

## [...]
  kestra:
    image: kestra/kestra:latest
    pull_policy: always
    # Note that this is meant for development only. Refer to the documentation for production deployments of Kestra which runs without a root user.
    user: "root"
    command: server standalone --worker-thread=128 --config /etc/config/application.yaml
    volumes:
      - kestra-data:/app/storage
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/kestra-wd:/tmp/kestra-wd
      - $PWD/application.yaml:/etc/config/application.yaml
    ports:
      - "8080:8080"
      - "8081:8081"
    depends_on:
      postgres:
        condition: service_started

Check out all of our available Docker image tags to see which one is best for your use case.

Networking in Docker Compose

The default docker-compose file doesn’t configure networking for the Kestra containers. This means that you won’t be able to access any services exposed via localhost on your local machine (e.g., another Docker container with a mapped port). Your machine and the Docker container operate on different networks. To use a locally exposed service from Kestra container, you can use the host.docker.internal hostname or 172.17.0.1. The host.docker.internal address allows you to reach your host machine’s services from Kestra’s container.

Alternatively, you can leverage Docker network. By default, your Kestra container is placed in a default network. You can add your custom services to the docker-compose.yml file provided by Kestra and use the services’ alias (keys from services) to reach them.

A better approach may be to create a new network (e.g., kestra_net) and add your services to it. Then, you can add this network to the networks section of the kestra service. With this configuration, you have access via localhost to all your exposed ports.

The example below shows how you can add iceberg-rest, minio, and mc (i.e., MinIO client) to your Kestra Docker Compose file.

Example

volumes:
  postgres-data:
    driver: local
  kestra-data:
    driver: local

networks:
  kestra_net:

services:
  postgres:
    image: postgres
    volumes:
      - postgres-data:/var/lib/postgresql
    environment:
      POSTGRES_DB: kestra
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: k3str4
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
      interval: 30s
      timeout: 10s
      retries: 10
    networks:
      kestra_net:

  iceberg-rest:
      image: tabulario/iceberg-rest
      ports:
        - 8181:8181
      environment:
        - AWS_ACCESS_KEY_ID=admin
        - AWS_SECRET_ACCESS_KEY=password
        - AWS_REGION=us-east-1
        - CATALOG_WAREHOUSE=s3://warehouse/
        - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO
        - CATALOG_S3_ENDPOINT=http://minio:9000
      networks:
        kestra_net:

  minio:
      image: minio/minio
      container_name: minio
      environment:
        - MINIO_ROOT_USER=admin
        - MINIO_ROOT_PASSWORD=password
        - MINIO_DOMAIN=minio
      networks:
        kestra_net:
          aliases:
            - warehouse.minio
      ports:
        - 9001:9001
        - 9000:9000
      command: ["server", "/data", "--console-address", ":9001"]

  mc:
      depends_on:
        - minio
      image: minio/mc
      container_name: mc
      networks:
        kestra_net:
      environment:
        - AWS_ACCESS_KEY_ID=admin
        - AWS_SECRET_ACCESS_KEY=password
        - AWS_REGION=us-east-1
      entrypoint: >
        /bin/sh -c "
        until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;
        /usr/bin/mc rm -r --force minio/warehouse;
        /usr/bin/mc mb minio/warehouse;
        /usr/bin/mc policy set public minio/warehouse;
        tail -f /dev/null
        "

  kestra:
      image: kestra/kestra:latest
      pull_policy: always
      entrypoint: /bin/bash
      # Note that this is meant for development only. Refer to the documentation for production deployments of Kestra which runs without a root user.
      user: "root"
      command:
        - -c
        - /app/kestra server standalone --worker-thread=128
      volumes:
        - kestra-data:/app/storage
        - /var/run/docker.sock:/var/run/docker.sock
        - /tmp/kestra-wd:/tmp/kestra-wd
      environment:
        KESTRA_CONFIGURATION: |
          datasources:
            postgres:
              url: jdbc:postgresql://postgres:5432/kestra
              driver-class-name: org.postgresql.Driver
              username: kestra
              password: k3str4
          kestra:
            server:
              basic-auth:
                username: admin
                password: kestra
            repository:
              type: postgres
            storage:
              type: minio
              minio:
                endpoint: http://minio
                port: 9000
                access-key: admin
                secret-key: password
                region: us-east-1
                bucket: warehouse
            queue:
              type: postgres
            tasks:
              tmp-dir:
                path: /tmp/kestra-wd/tmp
            url: http://localhost:8080/
      ports:
        - "8080:8080"
        - "8081:8081"
      depends_on:
        postgres:
          condition: service_started
      networks:
        kestra_net:

Finally, you can also use the host network mode for the kestra service. This makes your container use your host network, and you are then able to reach all your exposed ports. This means you have to change the services.kestra.environment.KESTRA_CONFIGURATION.datasources.postgres.url to jdbc:postgresql://localhost:5432/kestra. This is the easiest way reach all ports, but it can be a security risk.

See the example below using network_mode: host.

Example

volumes:
  kestra-data:
    driver: local

services:
  kestra:
    image: kestra/kestra:latest
    pull_policy: always
    entrypoint: /bin/bash
    network_mode: host
    environment:
      JAVA_OPTS: "--add-opens java.base/java.nio=ALL-UNNAMED"
      NODE_OPTIONS: "--max-old-space-size=4096"
      KESTRA_CONFIGURATION: |
        datasources:
          postgres:
            url: jdbc:postgresql://localhost:5432/kestra
            driver-class-name: org.postgresql.Driver
            username: kestra
            password: k3str4
        kestra:
          server:
            basic-auth:
              username: admin
              password: kestra
          anonymous-usage-report:
            enabled: true
          repository:
            type: postgres
          storage:
            type: local
            local:
              base-path: "/app/storage"
          queue:
            type: postgres
          tasks:
            tmp-dir:
              path: /tmp/kestra-wd/tmp
            scripts:
              docker:
                volume-enabled: true
            defaults: # Example demonstrating global pluginDefaults
              - type: io.kestra.plugin.airbyte.connections.Sync
                url: http://host.docker.internal:8000/
                username: airbyte
                password: password
          url: http://localhost:8080/
          variables:
            env-vars-prefix: "" # To avoid requiring KESTRA_ prefix on env vars

Postgres 16 not compatible with 17 error

By default, the Docker Compose template uses the latest image for PostgreSQL. However, if you initialized your Kestra database on an older version of PostgreSQL, you might encounter the following error:

The data directory was initialized by PostgreSQL version 16, which is not compatible with this version 17.0 (Debian 17.0-1.pgdg120+1).

To resolve this, you need to specify a specific tag for the PostgreSQL image in your Docker Compose file. In the example below, we specify 16 as the error above was originally initialized by version 16:

services:
  postgres:
    image: postgres:16

SIGILL in Java Runtime Environment on macOS M4 chip

Add the following environment variable to your Kestra container: -e JAVA_OPTS="-XX:UseSVE=0":

docker run --pull=always --rm -it -p 8080:8080 --user=root -e JAVA_OPTS="-XX:UseSVE=0" -v /var/run/docker.sock:/var/run/docker.sock -v /tmp:/tmp kestra/kestra:latest server local

To apply the same setting in a Docker Compose file:

services:
  kestra:
    image: kestra/kestra:latest
    environment:
      JAVA_OPTS: "-XX:UseSVE=0"

Kestra with server components in different services

Server components can run independently from each other. Each of them communicate through the database. The kestra server command starts each server component individually:

kestra server executor
kestra server worker
kestra server indexer
kestra server scheduler
kestra server webserver

For more details on Kestra server commands, check out the Server CLI documentation.

Here is an example Docker Compose configuration file running Kestra services with replicas on the Postgres database backend.

volumes:
  postgres-data:
    driver: local
  kestra-data:
    driver: local

services:
  postgres:
    image: postgres
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: kestra
      POSTGRES_USER: kestra
      POSTGRES_PASSWORD: k3str4
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
      interval: 30s
      timeout: 10s
      retries: 10

  kestra-scheduler:
    image: kestra/kestra:latest
    deploy:
      replicas: 2
    pull_policy: if_not_present
    user: "root"
    command: server scheduler
    volumes:
      - kestra-data:/app/storage
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/kestra-wd:/tmp/kestra-wd
    environment:
      KESTRA_CONFIGURATION: &common_configuration |
        datasources:
          postgres:
            url: jdbc:postgresql://postgres:5432/kestra
            driver-class-name: org.postgresql.Driver
            username: kestra
            password: k3str4
        kestra:
          server:
            basic-auth:
              username: "admin@kestra.io"
              password: kestra
          repository:
            type: postgres
          storage:
            type: local
            local:
              base-path: "/app/storage"
          queue:
            type: postgres
          tasks:
            tmp-dir:
              path: /tmp/kestra-wd/tmp
    ports:
      - "8082-8083:8081"
    depends_on:
      postgres:
        condition: service_started

  kestra-worker:
    image: kestra/kestra:latest
    deploy:
      replicas: 2
    pull_policy: if_not_present
    user: "root"
    command: server worker
    volumes:
      - kestra-data:/app/storage
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/kestra-wd:/tmp/kestra-wd
    environment:
      KESTRA_CONFIGURATION: *common_configuration
    ports:
      - "8084-8085:8081"
    depends_on:
      postgres:
        condition: service_started
  kestra-executor:
    image: kestra/kestra:latest
    deploy:
      replicas: 2
    pull_policy: if_not_present
    user: "root"
    command: server executor
    volumes:
      - kestra-data:/app/storage
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/kestra-wd:/tmp/kestra-wd
    environment:
      KESTRA_CONFIGURATION: *common_configuration
    ports:
      - "8086-8087:8081"
    depends_on:
      postgres:
        condition: service_started
  kestra-webserver:
    image: kestra/kestra:latest
    deploy:
      replicas: 1
    pull_policy: if_not_present
    user: "root"
    command: server webserver
    volumes:
      - kestra-data:/app/storage
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/kestra-wd:/tmp/kestra-wd
    environment:
      KESTRA_CONFIGURATION: *common_configuration
      KESTRA_URL: http://localhost:8080/
    ports:
      - "8080:8080"
      - "8081:8081"
    depends_on:
      postgres:
        condition: service_started

Was this page helpful?

InstallationRun Kestra with Docker – Single-Container Setup

InstallationDeploy Kestra on Kubernetes with Helm