​Troubleshooting

Something doesn't work as expected? Check out these common issues and fixes.

CrashLoopBackoff when restarting all pods

Question: "When I restart all Kubernetes pods at once, they get stuck in a CrashLoopBackoff for a number of minutes before eventually resolving—why does it happen?"

This is likely caused by Java startup behavior, which can consume a lot of resources and cause liveness probes to fail. Since Java loads many classes at startup, pods may restart multiple times before stabilizing. Setting the CPU limit to 2 times the request can improve startup time and resolve failing health checks.

Unprocessable execution

Sometimes, executions cannot be processed. In such cases, you can instruct Kestra to skip them.

Start the executor server (or the standalone server if not using a multi-component deployment) with a list of execution identifiers to skip:

kestra server executor --skip-executions 6FSPERUe1JwbYmMmdwRlgV,5iLGjTLOHAVGUGlsesFaMb

You can also skip executions at broader levels:

1. Flows — Skip all executions of one or more flows:
   sh

   kestra server executor "--skip-flows=tenant|namespace|flowA,tenant|namespace|flowB"
   

   Example:
   sh

   kestra server executor "--skip-flows=tenant|namespace|daily-data-sync"
   

2. Namespaces — Skip all executions within specific namespaces:
   sh

   kestra server executor "--skip-namespaces=tenant|myNamespaceA,tenant|myNamespaceB"
   

   Example:
   sh

   kestra server executor "--skip-namespaces=tenant|production-data"
   

3. Tenants — Skip all executions associated with specific tenants:
   sh

   kestra server executor "--skip-tenants=tenantA,tenantB"
   

   Example:
   sh

   kestra server executor "--skip-tenants=companyA"
   

Docker in Docker (DinD)

If you face issues using Docker in Docker (e.g., with Script tasks using the DOCKER runner), start troubleshooting by attaching to the DinD container:

docker run -it --privileged docker:dind sh

From there, use:

• docker logs <container_id> to view logs
• docker inspect <container_id> to get environment, network, and configuration details

These commands help diagnose misconfigurations.

Docker in Docker using Helm charts

On some Kubernetes deployments, using DinD with our default Helm charts can result in errors such as:

Device "ip_tables" does not exist.
ip_tables              24576  4 iptable_raw,iptable_mangle,iptable_nat,iptable_filter
modprobe: can't change directory to '/lib/modules': No such file or directory
error: attempting to run rootless dockerd but need 'kernel.unprivileged_userns_clone' (/proc/sys/kernel/unprivileged_userns_clone) set to 1

To fix this, launch the DinD container as root by setting the following values:

dind:
  image:
    tag: dind
  args:
    - --log-level=fatal
  securityContext:
    runAsUser: 0
    runAsGroup: 0

securityContext:
  runAsUser: 0
  runAsGroup: 0

DinD on a Mac with Apple silicon (ARM)

If you see errors like:

java.io.IOException: com.sun.jna.LastErrorException: [111] Connection refused

it may be caused by running Docker in Docker on ARM-based Macs.

Try using an embedded Docker server as shown below:

tmp directory errors ("No such file or directory")

If you encounter errors such as "No such file or directory" related to the tmp directory, it usually means the directory is not mounted correctly.

In your docker-compose.yml, ensure the tmpDir path matches the mounted volume:

kestra:
  tasks:
    tmpDir:
      path: /home/kestra/tmp

Volume example:

volumes:
  - kestra-data:/app/storage
  - /var/run/docker.sock:/var/run/docker.sock
  - /home/kestra:/home/kestra

This ensures Kestra can properly access the tmp directory.