Process Task Runner – Run Tasks as Local Processes

Run tasks as local processes.

Run tasks locally with the Process runner

The following example shows a Shell script configured with the Process task runner, which runs a Shell command as a child process on the Kestra host:

id: process_task_runner
namespace: company.team

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    commands:
      - echo "Hello World!"

The Process task runner does not require any additional configuration beyond the type property.

Benefits

The Process task runner is useful when you need to access local files or take advantage of locally configured software libraries and virtual environments.

Combining task runners with Worker Groups

You can combine the Process task runner with Worker Groups to run tasks on dedicated servers that have specific software libraries or configurations. This combination allows you to leverage the compute resources of your Worker Groups while running tasks as local processes, without the overhead of containerization.

The following example demonstrates how to combine the Process task runner with Worker Groups to fully leverage the GPU resources of a dedicated server:

id: python_on_gpu
namespace: company.team

tasks:
  - id: gpu_intensive_ai_workload
    type: io.kestra.plugin.scripts.python.Commands
    namespaceFiles:
      enabled: true
    commands:
      - python main.py
    workerGroup:
      key: gpu
    taskRunner:
      type: io.kestra.plugin.core.runner.Process