Available on: >= 0.16.0

How Task Runners can help with resource allocation and environment management.

Docker in development, Kubernetes in production

Many Kestra users develop their scripts locally in Docker containers and then run the same code in a production environment as Kubernetes pods. Thanks to the taskRunner property, setting this up is a breeze. Below is an example showing how you can combine taskDefaults with the taskRunner property to use Docker in the development environment and Kubernetes in production β€” all without changing anything in your code.

  1. Development namespace/tenant/instance:
    yaml
    taskDefaults:
      - type: io.kestra.plugin.scripts
        values:
          taskRunner:
            type: io.kestra.plugin.scripts.runner.docker.DockerTaskRunner
            pullPolicy: IF_NOT_PRESENT # in dev, only pull the image when needed
            cpu:
              cpus: 1
            memory:
              memory: 512Mi
    
  2. Production namespace/tenant/instance:
    yaml
    taskDefaults:
      - type: io.kestra.plugin.scripts
        values:
          taskRunner:
            type: io.kestra.plugin.kubernetes.runner.KubernetesTaskRunner
            namespace: kestra-prd
            pullPolicy: ALWAYS # Always pull the latest image in production
            config:
              username: "{{ secret('K8S_USERNAME') }}"
              masterUrl: "{{ secret('K8S_MASTER_URL') }}"
              caCert: "{{ secret('K8S_CA_CERT') }}"
              clientCert: "{{ secret('K8S_CLIENT_CERT') }}"
              clientKey: "{{ secret('K8S_CLIENT_KEY') }}"
            resources: # can be overriden by a specific task if needed
              request: # The resources the container is guaranteed to get
                cpu: "500m" # Request 1/2 of a CPU (500 milliCPU)
                memory: "256Mi" # Request 256 MB of memory
    

Centralized configuration management

The combination of taskDefaults and taskRunner properties allows you to centrally manage your task runner configuration. For example, you can use taskDefaults on a namespace level to centrally manage your AWS credentials for the AwsBatchTaskRunner plugin.

yaml
taskDefaults:
  - type: io.kestra.plugin.aws
    values:
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
      region: "us-east-1"

This configuration will apply to all components of the AWS plugin incl. tasks, triggers and task runners. This includes the io.kestra.plugin.aws.runner.AwsBatchTaskRunner.

Documentation and autocompletion

Each task runner is a plugin with its own icon, documentation, and schema to validate its properties. The built-in code editor in the Kestra UI provides autocompletion and syntax validation for all runner properties, and when you click on the runner's name in the editor, you can see its documentation on the right side of the screen.

docker_runner

Full customization: create your own Task Runner

You can create a custom task runner plugin for your specific environment, build it as a JAR file, and add that file to the plugins directory. Once you restart Kestra, your custom runner plugin will be available on any script task in the system.

You can contribute your custom task runner plugin to the Kestra community, and if you do, we'll be happy to include it on the plugins page.

Was this page helpful?