DocsDeveloper GuidePython, R, Node.js and Shell ScriptsDOCKER and PROCESS runners

DOCKER and PROCESS runners​D​O​C​K​E​R and ​P​R​O​C​E​S​S runners

Was this page helpful?

Kestra supports two runners for scripting tasks: DOCKER and PROCESS.

You can configure your scripts to run either in local processes or in Docker containers by using the runner property:

  1. By default all scripting tasks run in isolated containers using the DOCKER runner.
  2. Setting the runner property to PROCESS will execute your task in a local process on the worker without relying on Docker for container isolation.

runner: DOCKER

Docker is the default option for all script tasks. There are many arguments that can be provided here, including credentials to private Docker registries:

yaml
id: python_in_container
namespace: dev

tasks:
  - id: wdir
    type: io.kestra.core.tasks.flows.WorkingDirectory
    tasks:
      - id: cloneRepository
        type: io.kestra.plugin.git.Clone
        url: https://github.com/kestra-io/examples
        branch: main

      - id: gitPythonScripts
        type: io.kestra.plugin.scripts.python.Commands
        warningOnStdErr: false
        commands:
          - python scripts/etl_script.py
        runner: DOCKER
        docker:
          image: annageller/kestra:latest
          config: |
            {
              "auths": {
                  "https://index.docker.io/v1/": {
                      "username": "annageller",
                      "password": "{{ secret('DOCKER_PAT') }}"
                  }
              }
            }
      - id: output
        type: io.kestra.core.tasks.storages.LocalFiles
        outputs:
          - "*.csv"
          - "*.parquet"

Head over to the Secrets section to learn more about secrets in Kestra.

runner: PROCESS

The PROCESS runner is useful if your Kestra instance is running locally without Docker and you want to access your local files and environments, for example, to take advantage of locally configured Conda virtual environments.

yaml
id: local_python_script
namespace: dev

tasks:
  - id: conda_example
    type: io.kestra.plugin.scripts.python.Commands
    runner: PROCESS
    beforeCommands:
      - conda activate myCondaEnv
    commands:
          - python /Users/you/scripts/etl_script.py

Running scripts in a local process is particularly beneficial when using remote Worker Groups. The example below ensures that a script will be picked up only by Kestra workers that have been started with the key gpu, effectively delegating processing of scripts that require demanding computational requirements to the right server, rather than running them directly in a local container:

yaml
id: gpu_task
namespace: dev

tasks:
  - id: gpu
    type: io.kestra.plugin.scripts.python.Commands
    runner: PROCESS
    commands:
          - python ml_on_gpu.py
    workerGroup:
      key: gpu