Kestra Task Runners – Offload and Isolate Compute
Task Runners is an extensible, pluggable system capable of executing your tasks in arbitrary remote environments.
Offload and isolate compute with task runners
Many data processing tasks are computationally intensive and require a lot of resources (such as CPU, GPU, and memory). Instead of provisioning always-on servers, Task Runners can execute your code on dynamically provisioned compute instances in the cloud, such as AWS ECS Fargate, Azure Batch, Google Batch, auto-scaled Kubernetes clusters, and more.
All you have to do to offload your task execution to a remote environment is to specify the taskRunner type in your task configuration. Each type of a task runner is a plugin with its own schema. The built-in code editor provides documentation, autocompletion, and syntax validation for all task runner plugin properties to ensure correctness, standardization, and consistency.
Note that some task runner plugins are available only in the Enterprise Edition. If you want to try them out, please reach out. See Open Source vs Enterprise to compare editions.
Task Runner Overview – Capabilities and Supported Plugins
Learn about Kestra Task Runners capabilities and supported plugins for executing tasks in diverse environments.
Task Runner Benefits – Resource Control and Deployment Flexibility
Explore the benefits of using Task Runners in Kestra for isolated execution, resource control, and deployment flexibility.
Task Runners vs Worker Groups – When to Use Each
Learn when to use Task Runners versus Worker Groups in Kestra for optimal compute resource management and isolation.
Task Runner Types – Choose the Right Execution Backend
Explore the different types of Task Runners available in Kestra, including Docker, Process, Kubernetes, and cloud-based runners.
Was this page helpful?