Choosing Between Task Runners and Worker Groups in Kestra

Worker groups decide which worker runs a task, and task runners decide where a script task’s code runs. One task can use both.

How they fit together

Every task in Kestra runs on a worker. Two independent settings decide where the work happens:

• A Worker Group decides which worker picks up the task, and so which network, region, or host the task starts from. You route a task to a Worker Group with workerSelector.
• A task runner decides where a script task’s code runs once a worker has picked it up: as a process on the worker host, in a Docker container on the worker host, or as a job on remote compute such as Kubernetes or AWS Batch. You choose one with taskRunner.

A task can set either one, both, or neither. A task with neither runs in the default Worker Group. If it is a script task, it also runs in a Docker container on that worker’s host, because Docker is the default task runner.

The workerSelector routes the task first. If no Worker Group serves the matching Worker Queue, the task follows its fallback before any task runner starts.

What a remote task runner does

When a task uses a remote task runner, the worker that picked up the task stays in charge of it for the whole run:

1. The worker renders the commands, stages input files and namespace files, and submits the job through the platform’s API.
2. The job runs on the remote compute, and the worker streams its logs back to Kestra.
3. When the job ends, the worker collects the declared output files and reports the result.

This has two consequences:

• The job is submitted from the worker’s network. The worker needs network access to the platform’s API, such as the Kubernetes API server or the AWS Batch endpoint, plus credentials for it.
• The task holds a worker thread until the remote job finishes, because each worker thread runs one task at a time. A remote task runner adds compute for the script. It does not add worker capacity.

Choose by what the task needs

Key differences

Example: combine a Worker Group and a task runner

The following flow trains a model on a Kubernetes cluster whose API server is only reachable from region B:

id: train_in_region_b
namespace: company.team

tasks:
  - id: train
    type: io.kestra.plugin.scripts.python.Commands
    namespaceFiles:
      enabled: true
    containerImage: ghcr.io/kestra-io/pydata:latest
    workerSelector:
      tags: [region-b]
    taskRunner:
      type: io.kestra.plugin.ee.kubernetes.runner.Kubernetes
      namespace: ml-jobs
      config:
        masterUrl: https://k8s.region-b.internal:6443
        caCertData: "{{ secret('K8S_CA_CERT_DATA') }}"
        oauthToken: "{{ secret('K8S_OAUTH_TOKEN') }}"
    commands:
      - python train.py

A worker from a Worker Group subscribed to the Worker Queue tagged region-b picks up the task. That worker calls the API server at masterUrl, and the pod runs on the region B cluster.

Change one setting and the behavior changes:

• Without workerSelector, the task runs in the default Worker Group, so the default workers need network access to the region B API server.
• Without taskRunner, the script runs in a Docker container on the region B worker host.

Next steps

• To create Worker Queues and Worker Groups, register workers, and set fallbacks and capacity, see Worker Groups.
• To compare the available runners, see task runner types.
• To configure a task runner on a script task, see Task runners for scripts.