Task Runner Benefits​Task ​Runner ​Benefits

Available on: Open Source EditionEnterprise Edition>= 0.18.0

How Task Runners can help with resource allocation and environment management.

Docker in development, Kubernetes in production

Many Kestra users develop their scripts locally in Docker containers and then run the same code in a production environment as Kubernetes pods. Thanks to the taskRunner property, setting this up is a breeze. Below is an example showing how you can combine pluginDefaults with the taskRunner property to use Docker in the development environment and Kubernetes in production โ€” all without changing anything in your code.

  1. Development namespace/tenant/instance:
    yaml
    pluginDefaults:
      - type: io.kestra.plugin.scripts
        values:
          taskRunner:
            type: io.kestra.plugin.scripts.runner.docker.Docker
            pullPolicy: IF_NOT_PRESENT # in dev, only pull the image when needed
            cpu:
              cpus: 1
            memory:
              memory: 512Mi
    
  2. Production namespace/tenant/instance:
    yaml
    pluginDefaults:
      - type: io.kestra.plugin.scripts
        values:
          taskRunner:
            type: io.kestra.plugin.ee.kubernetes.runner.Kubernetes
            namespace: company.team
            pullPolicy: ALWAYS # Always pull the latest image in production
            config:
              username: "{{ secret('K8S_USERNAME') }}"
              masterUrl: "{{ secret('K8S_MASTER_URL') }}"
              caCert: "{{ secret('K8S_CA_CERT') }}"
              clientCert: "{{ secret('K8S_CLIENT_CERT') }}"
              clientKey: "{{ secret('K8S_CLIENT_KEY') }}"
            resources: # can be overriden by a specific task if needed
              request: # The resources the container is guaranteed to get
                cpu: "500m" # Request 1/2 of a CPU (500 milliCPU)
                memory: "256Mi" # Request 256 MB of memory
    

Centralized configuration management

The combination of pluginDefaults and taskRunner properties allows you to centrally manage your task runner configuration. For example, you can use pluginDefaults on a namespace level to centrally manage your AWS credentials for the Batch task runner plugin.

yaml
pluginDefaults:
  - type: io.kestra.plugin.ee.aws.runner.Batch
    values:
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
      region: "us-east-1"

Documentation and autocompletion

Each task runner is a plugin with its own icon, documentation, and schema to validate its properties. The built-in code editor in the Kestra UI provides autocompletion and syntax validation for all runner properties, and when you click on the runner's name in the editor, you can see its documentation on the right side of the screen.

docker_runner

Full customization: create your own Task Runner

You can create a custom task runner plugin for your specific environment, build it as a JAR file, and add that file to the plugins directory. Once you restart Kestra, your custom runner plugin will be available on any script task in the system.

Was this page helpful?