Task Runner Overview
Available on: Open Source EditionEnterprise Edition>= 0.18.0
Understand the capabilities of Task Runners and the plugins that support them.
Overview
Task Runners provide a flexible and efficient way to execute compute-intensive workloads across different environments. Whether you’re running scripts locally, on Kubernetes, or on cloud platforms like AWS, Azure, or Google Cloud, Task Runners ensure consistent, isolated, and configurable task execution.
Key capabilities
The table below outlines the main capabilities of Task Runners in Kestra.
| Capability | Description |
|---|---|
| Fine-grained resource allocation | Gain full control over compute resources — allocate the precise amount of CPU, memory, or GPU to individual tasks. |
| Flexible deployment patterns | Deploy tasks across diverse environments, including AWS ECS Fargate, Azure Batch, Google Batch, Kubernetes, and more. You can even mix different runners within a single workflow. |
| No vendor lock-in | Built on a modular plugin system, Task Runners let you run workloads on any cloud or on-prem infrastructure — without being tied to a specific provider. |
| Task isolation | Each task runs in a fully isolated container environment, preventing conflicts and ensuring consistent performance. |
| Development-to-production consistency | Develop locally using Docker containers and seamlessly deploy the same code to production in Kubernetes or cloud environments — just by changing one property. |
| Centralized configuration management | Define and manage runner configurations globally using pluginDefaults. This allows you to govern credentials and environment settings at the namespace or organization level. |
| Built-in documentation and validation | Each Task Runner plugin includes a schema. The Kestra code editor offers inline documentation, autocompletion, and syntax validation for every property, ensuring correctness and standardization. |
| No code changes required | Move between environments — from local to cloud — without altering your business logic or code. |
| Fully customizable | Extend functionality by developing your own Task Runner plugin tailored to your infrastructure and deployment requirements. |
Supported plugins
Task Runners are primarily used in tasks from the Script Plugin and its related sub-plugins. These tasks support execution of custom scripts or command sets via the taskRunner property.
Supported script-based plugins
- Python
- Node.js
- Go
- Shell
- PowerShell
- R
- Julia
- Ruby
- Deno
- Lua
- Bun
- PHP
- Perl
- Groovy
- dbt
- SQLMesh
- Ansible
- Terraform
- Modal
- AWS CLI
- GCloud CLI
- Azure CLI
Whenever you see a task capable of executing a script or a series of commands, it’s a script-based task that can leverage a taskRunner to define where and how that task runs.
Was this page helpful?