Task Runner Benefits
Available on: Open Source EditionEnterprise Edition>= 0.18.0
How Task Runners can help with resource allocation and environment management.
Docker in development, Kubernetes in production
Many Kestra users develop their scripts locally in Docker containers and then run the same code in a production environment as Kubernetes pods. Thanks to the taskRunner
property, setting this up is a breeze. Below is an example showing how you can combine pluginDefaults
with the taskRunner
property to use Docker in the development environment and Kubernetes in production โ all without changing anything in your code.
- Development namespace/tenant/instance:yaml
pluginDefaults: - type: io.kestra.plugin.scripts values: taskRunner: type: io.kestra.plugin.scripts.runner.docker.Docker pullPolicy: IF_NOT_PRESENT # in dev, only pull the image when needed cpu: cpus: 1 memory: memory: 512Mi
- Production namespace/tenant/instance:yaml
pluginDefaults: - type: io.kestra.plugin.scripts values: taskRunner: type: io.kestra.plugin.ee.kubernetes.runner.Kubernetes namespace: company.team pullPolicy: ALWAYS # Always pull the latest image in production config: username: "{{ secret('K8S_USERNAME') }}" masterUrl: "{{ secret('K8S_MASTER_URL') }}" caCert: "{{ secret('K8S_CA_CERT') }}" clientCert: "{{ secret('K8S_CLIENT_CERT') }}" clientKey: "{{ secret('K8S_CLIENT_KEY') }}" resources: # can be overriden by a specific task if needed request: # The resources the container is guaranteed to get cpu: "500m" # Request 1/2 of a CPU (500 milliCPU) memory: "256Mi" # Request 256 MB of memory
Note how the containerImage
property is not included in the taskRunner
configuration but as a generic property available to any scripting task. This makes the configuration more flexible as usually the container image changes more often than the standard runner configuration. For instance, the dbt plugin may need a different image than the generic Python plugin, but the runner configuration can stay the same.
Centralized configuration management
The combination of pluginDefaults
and taskRunner
properties allows you to centrally manage your task runner configuration. For example, you can use pluginDefaults
on a namespace level to centrally manage your AWS credentials for the Batch
task runner plugin.
pluginDefaults:
- type: io.kestra.plugin.ee.aws.runner.Batch
values:
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
region: "us-east-1"
Documentation and autocompletion
Each task runner is a plugin with its own icon, documentation, and schema to validate its properties. The built-in code editor in the Kestra UI provides autocompletion and syntax validation for all runner properties, and when you click on the runner's name in the editor, you can see its documentation on the right side of the screen.
Full customization: create your own Task Runner
You can create a custom task runner plugin for your specific environment, build it as a JAR file, and add that file to the plugins
directory. Once you restart Kestra, your custom runner plugin will be available on any script task in the system.
Was this page helpful?