Task Runners are now Generally Available and Ready to Handle Your Most Demanding Workflows

August 8 2024News & Product Updates

Anna Geller

Table of Contents

Task Runners are now Generally Available and Ready to Handle Your Most Demanding Workflows

Run your code anywhere with dynamically-provisioned resources.

We are thrilled to announce the general availability of Task Runners, a major addition to Kestra’s orchestration capabilities, allowing you to offload resource-intensive tasks to on-demand compute services. Task runners guarantee that your workflows have enough resources while reducing compute costs.

Why Task Runners?

Many data processing tasks are computationally intensive and require a lot of resources (such as CPU, GPU, and memory). Instead of provisioning always-on servers, Task Runners can execute your code on dynamically provisioned containers in the cloud, such as AWS ECS Fargate, Azure Batch, Google Batch, Google Cloud Run, auto-scaled Kubernetes clusters, and more.

What are Task Runners

Task Runners is an extensible ecosystem of plugins capable of executing your tasks in arbitrary remote environments. All you have to do to offload data processing to a remote environment is to specify the taskRunner type in your task configuration.

You can either build a custom plugin to run your tasks in any environment you wish, or you can use one of the managed plugins offered in Kestra Enterprise or Kestra Cloud, such as AWS Batch, Azure Batch, Google Batch, Google Cloud Run, or Kubernetes.

task_runner_plugins

From Beta to General Availability

We introduced task runners in Beta in Kestra 0.16.0, and since then, we’ve been improving their performance, stability, and usability. Among others, we’ve added the capability to terminate remote workers when the execution is canceled from the UI, integrated Task Runners into additional CLI and script plugins, improved file handling and recovery from failures in remote compute environments, and documented the feature extensively.

Thanks to feedback from over 80 users and many enhancements and bug fixes, Task Runners are now generally available and ready for production use at scale. We are grateful to all our Beta testers for their valuable input and suggestions.

“Our pipelines were faster in GCP Batch service compared to Cloud Run Jobs, and they used even less memory and CPU. I attribute this to the simplified code, and simplified design of how Kestra Task Runners only poll VMs as a whole.” — Jack P., Data Engineer at Foundation Direct

Key Benefits of Task Runners

Fine-grained resource allocation

Task Runners empower you with fine-grained resource allocation, ensuring that you can precisely adjust the CPU, memory, and GPU needed for any given task. With built-in support for multiple cloud providers and the ability to build custom plugins for any environment, Task Runners give you full flexibility to evolve your infrastructure as your needs change over time.

Fast development with autocompletion, built-in documentation and blueprints

Thanks to the built-in documentation and autocompletion, building workflows with Task Runners is easy and fast. When you add a specific taskRunner to your workflow in the Code Editor, its documentation appears on the right side of the screen, providing immediate access to all available properties and usage examples. Additionally, the syntax validation helps you gain confidence that your task runner configuration is correct before you run it.

To help you get started, we’ve created several pre-built workflow templates. Many of them include automated deployment of IAM roles and other required Cloud services to quickly set up the Task Runner that matches your environment. The blueprint example below automates the setup of an AWS Batch environment to run multiple containerized Python scripts on AWS ECS Fargate.

task_runner_blueprints

From development to production

One of the key benefits of Task Runners is their ability to run the same business logic in different environments without changing anything in your code. This significantly speeds up the development process and simplifies the transition from development to staging and production environments.

Consistent API with centralized configuration

Whether you are developing locally in Docker or running production workloads in Kubernetes, Task Runners offer a consistent API, requiring no changes to your business logic code. Thanks to pluginDefaults, you can manage your task runner configuration and credentials in a single place for each environment without code duplication.

Check the video below summarizing the key benefits of Task Runners.

Next steps

Embrace the scalability of dynamically-provisioned resources with Task Runners, now fully equipped to handle your most demanding data processing workflows.

To help you get started, we prepared extensive How-To Guides, Blueprints and Video Tutorials on how to use Task Runners. The videos linked below will guide you through the process of setting up a Task Runner for your chosen cloud provider.