You are viewing the docs for v1.3, which is not the latest release.
View latest

Diagnose and resolve common Kestra deployment and runtime issues

For the complete documentation index, see llms.txt. For a full content snapshot, see llms-full.txt. Append .md to any kestra.io/docs/* URL for plain Markdown.

Common issues and fixes for Kestra deployments.

CrashLoopBackoff when restarting all pods

Question: "When I restart all Kubernetes pods at once, they get stuck in a CrashLoopBackoff for a number of minutes before eventually resolving — why does it happen?"

This is likely caused by Java startup behavior, which can consume a lot of resources and cause liveness probes to fail. Since Java loads many classes at startup, pods may restart multiple times before stabilizing. Setting the CPU limit to 2 times the request can improve startup time and resolve failing health checks.

Unprocessable execution

Sometimes, executions cannot be processed. In such cases, you can instruct Kestra to skip them.

Start the executor server (or the standalone server if not using a multi-component deployment) with a list of execution identifiers to skip:

shkestra server executor --skip-executions 6FSPERUe1JwbYmMmdwRlgV,5iLGjTLOHAVGUGlsesFaMb

You can also skip executions at broader levels:

  1. Flows — Skip all executions of one or more flows:
    shkestra server executor "--skip-flows=tenant|namespace|flowA,tenant|namespace|flowB"
    Example:
    shkestra server executor "--skip-flows=companyA|production-data|daily-data-sync"
  1. Namespaces — Skip all executions within specific namespaces:
    shkestra server executor "--skip-namespaces=tenant|myNamespaceA,tenant|myNamespaceB"
    Example:
    shkestra server executor "--skip-namespaces=companyA|production-data"
  1. Tenants — Skip all executions associated with specific tenants:
    shkestra server executor "--skip-tenants=tenantA,tenantB"
    Example:
    shkestra server executor "--skip-tenants=companyA"

Docker in Docker (DinD)

If you face issues using Docker in Docker (e.g., with Script tasks using the DOCKER runner), start troubleshooting by attaching to the DinD container:

shdocker run -it --privileged docker:dind sh

From there, use:

  • docker logs <container_id> to view logs
  • docker inspect <container_id> to get environment, network, and configuration details

These commands help diagnose misconfigurations.

Docker in Docker using Helm charts

On some Kubernetes deployments, using DinD with our default Helm charts can result in errors such as:

bashDevice "ip_tables" does not exist.
ip_tables              24576  4 iptable_raw,iptable_mangle,iptable_nat,iptable_filter
modprobe: can't change directory to '/lib/modules': No such file or directory
error: attempting to run rootless dockerd but need 'kernel.unprivileged_userns_clone' (/proc/sys/kernel/unprivileged_userns_clone) set to 1

To fix this, switch DinD to insecure (privileged) mode by setting the following values:

yamldind:
  mode: 'insecure'
  base:
    insecure:
      image:
        tag: dind
      args:
        - --log-level=fatal
      securityContext:
        runAsUser: 0
        runAsGroup: 0

DinD on a Mac with Apple silicon (ARM)

If you see errors like:

plaintextjava.io.IOException: com.sun.jna.LastErrorException: [111] Connection refused

it may be caused by running Docker in Docker on ARM-based Macs.

Try using an embedded Docker server as shown below:

Example docker-compose.yml
yaml## volumes omitted for brevity
services:
  postgres:
    image: postgres
    # ...
  dind:
    image: docker:dind
    privileged: true
    environment:
      DOCKER_HOST: unix://dind/docker.sock
    command:
      - --log-level=fatal
    volumes:
      - dind-socket:/dind
      - tmp-data:/tmp/kestra-wd

  kestra:
    image: kestra/kestra:latest
    entrypoint: /bin/bash
    user: "root" # dev only — not for production
    command:
      - -c
      - /app/kestra server standalone --worker-thread=128
    volumes:
      - kestra-data:/app/storage
      - dind-socket:/dind
      - tmp-data:/tmp/kestra-wd
    environment:
      KESTRA_CONFIGURATION: |
        datasources:
          postgres:
            url: jdbc:postgresql://postgres:5432/kestra
            driver-class-name: org.postgresql.Driver
            username: kestra
            password: k3str4
        kestra:
          storage:
            type: local
            local:
              base-path: "/app/storage"
          queue:
            type: postgres
          tasks:
            tmp-dir:
              path: /tmp/kestra-wd/tmp
    ports:
      - "8080:8080"
      - "8081:8081"

tmp directory errors ("No such file or directory")

If you encounter errors such as "No such file or directory" related to the tmp directory, it usually means the directory is not mounted correctly.

In your docker-compose.yml, ensure the tmpDir path matches the mounted volume:

yamlkestra:
  tasks:
    tmpDir:
      path: /home/kestra/tmp

Example volume configuration:

yamlvolumes:
  - kestra-data:/app/storage
  - /var/run/docker.sock:/var/run/docker.sock
  - /home/kestra:/home/kestra

This ensures Kestra can properly access the tmp directory.