​Docker ​Task ​Runner

Run tasks as Docker containers.

How to use the Docker task runner

Here is an example using the Docker task runner executing the commands in a Docker container:

id: docker_script_runner
namespace: company.team

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    containerImage: centos
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
      cpu:
        cpus: 1
    commands:
      - echo "Hello World!"

Once you specify the taskRunner type, you get the autocompletion and validation for the runner-specific properties. In the example above, the task allocates 1 CPU to the container.

Docker task runner properties

The only property required by the taskRunner is the containerImage property that needs to be set on the script task. The image can be from a public or private registry.

Additionally, using the Docker task runner you can configure memory allocation, volumes, environment variables, and more. For a full list of properties available in the Docker task runner, check the Docker plugin documentation or explore the same in the built-in Code Editor in the Kestra UI.

Task runner behavior in a failure scenario

Generally speaking, each task runner container initiated by Kestra will continue running until the task completes, even if the Kestra worker is terminated (e.g. due to a crash). However, there are some caveats to be aware of depending on how Kestra and the task runner are deployed.

Kestra running in a Docker container, Task Runner running in DinD

When Kestra runs in a Docker container and uses DinD for task runners, terminating the Kestra container will also terminate the DinD container and any running task containers inside DinD. No container is automatically restarted.

Kestra running in Kubernetes, Task Runner running in DinD

When Kestra and DinD are deployed in the same pod in a Kubernetes environment, the pod will be restarted if Kestra Worker fails. This ensures that the DinD container and any task runner containers are also restarted.

Kestra deployed with Docker-Compose, Task Runner running in DinD

When using Docker-Compose, Kestra and DinD containers can be managed independently. Restarting the Kestra container does NOT automatically restart the DinD container. Therefore, task runners running inside DinD may continue running even if Kestra is restarted.

Insecure Registry

The Docker task runner supports insecure registries. Prior to using this feature, ensure the insecure registry is configured on the host machine that your Kestra server is running on.

For example, if you have the insecure registry 10.10.1.5:5000, please add the following configuration to /etc/docker/daemon.json and then restart your Docker daemon.

{
  "insecure-registries" : ["10.10.1.5:5000"]
}

This can then be used in a flow with the following YAML

id: docker_example
namespace: demo

tasks:
  - id: my_command
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
      config: |
        {
          "insecure-registries" : ["10.10.1.5:5000"]
        }
    containerImage: 10.10.1.5:5000/my-image
    commands:
      - echo "Hello World!"