Task Runners
Available on: >= 0.16.0
Task Runners are plugins capable of executing your tasks in remote environments.
Many data processing tasks are computationally intensive and require a lot of resources (such as CPU, GPU, and memory). Instead of provisioning always-on servers, Task Runners can run your code on dynamically provisioned compute instances in the cloud, such as AWS ECS Fargate, Azure Batch, Google Batch, auto-scaled Kubernetes clusters, and more.
Task Runner Overview
Task Runner capabilities and supported plugins.
Task Runner Benefits
How Task Runners can help with resource allocation and environment management.
Task Runner Use Cases
Use cases that can benefit from Task Runners.
Task Runner vs. Worker Group
Find out when to use task runners or worker groups.
Process Task Runner
Run tasks as local processes.
Docker Task Runner
Run tasks as Docker containers.
Kubernetes Task Runner
Run tasks as Kubernetes pods.
AWS Batch Task Runner
Run tasks as AWS ECS Fargate or EC2 containers using AWS Batch.
Azure Batch Task Runner
Run tasks as containers on Azure Batch VMs.
Google Batch Task Runner
Run tasks as containers on Google Cloud VMs.
Google Cloud Run Task Runner
Run tasks as containers on Google Cloud Run.
Task Runner Types
This section lists all task runners available in Kestra.