Long running and intensive processes on Kubernetes
Schedule long running and intensive processes with Kestra on Kubernetes.
Introduction
Long running tasks hold strong importance in the world of automation. They can range from data processing, machine learning, and data analytics to batch processing, ETL, and more.
While these tasks are essential for business operations, they can be resource-intensive and time-consuming while requiring specific hardware. To execute these tasks efficiently, you need a robust and scalable infrastructure that can handle the workload effectively.
Kestra offers various task execution solutions such as Docker, local processes, and Kubernetes. See Task Runners for more details.
In this guide, we will focus on executing long-running tasks on Kubernetes using Kestra.
Kubernetes pods are a great fit due to the control and flexibility they provide. With Kubernetes, you can precisely define resource requirements, permissions, namespace, handle workload identity, and ensure proper networking for your tasks. Pods can also easily access other Kubernetes services hosted on the cluster such as databases, storage and applications.
As an exmample, we will use a dbt job to demonstrate how Kestra can help you execute complex tasks on Kubernetes with resource requirements.
Schedule task in a Kubernetes pod using podCreate
Kestra's podCreate task allows you to launch a Kubernetes pod directly by providing the complete Kubernetes YAML configuration as an input. This gives you full control over the pod’s specifications, such as CPU, memory, image, or node selector.
Here is an example of a dbt job that runs on Kubernetes using Kestra:
id: my-dbt-job
namespace: dev
tasks:
- id: dbt-command
type: io.kestra.plugin.kubernetes.PodCreate
# Retry the task if it fails
retry:
behavior: RETRY_FAILED_TASK
maxAttempt: 2
type: constant
interval: PT5M
warningOnRetry: true
namespace: kestra
# Define the commands to run
inputFiles:
dbt-commands.sh: |
#!/bin/bash
# Exit on error
set -eo pipefail
# Clone the dbt example repository
git clone --depth 1 https://github.com/dbt-labs/jaffle_shop_duckdb.git --branch duckdb --single-branch
# Copy the dbt example repository to the working directory
cp -a jaffle_shop_duckdb/. .
# dbt commands to run
dbt deps
dbt build
# Define the pod specification using the Kubernetes YAML syntax
spec:
restartPolicy: Never
containers:
- name: dbt-duckdb
image: ghcr.io/kestra-io/dbt-duckdb:latest
# Specify resource requirements
resources:
request:
cpu: "300m"
memory: "500Mi"
# Run the script in the container
command:
- "/bin/bash"
- "{{workingDir}}/dbt-commands.sh"
# Node selector to run the pod on a specific node
nodeSelector: {}
This flow will:
- create a Kubernetes pod in the
kestra
namespace with the specified resource requirements: 300m CPU and 500Mi memory - clone the dbt example repository inside the pod
- run the dbt seed and build commands
At the end of the execution, the pod is deleted, and the logs remain available in the Kestra UI.
Embrace Kestra versality with Kubernetes Task Runners
While podCreate provides deep control, it takes aways all the benefits of Kestra's rich plugins ecosystem, dbt plugin in this case. Also it can be cumbersome to manage complex Kubernetes pod YAML specification for each task, especially when you have multiple commands to run.
To leverage the best of both worlds, Kestra’s Task Runners allow you to define workflows that benefit from Kestra's plugin system, enabling you to use familiar plugins while still leveraging the flexibility of Kubernetes to secure and scale the tasks effectively.
Also it gives you the flexibility to test your task easily before deploying it on Kubernetes using Task Runner types like Docker or Process.
The same example would look like this using a Task Runner:
id: dbt-task-runner
namespace: dev
tasks:
- id: dbt_build
type: io.kestra.plugin.dbt.cli.DbtCLI
taskRunner:
type: io.kestra.plugin.ee.kubernetes.runner.Kubernetes
namespace: kestra
resources:
request:
cpu: "300m"
memory: "500Mi"
containerImage: ghcr.io/kestra-io/dbt-duckdb:latest
commands:
- git clone --depth 1 https://github.com/dbt-labs/jaffle_shop_duckdb.git --branch duckdb --single-branch
- cp -a jaffle_shop_duckdb/. .
- dbt deps
- dbt build
In case we want to easily test this flow on a local Kestra instance, we would just have to change the taskRunner
type to io.kestra.plugin.scripts.runner.docker.Docker
:
id: dbt-task-runner
namespace: dev
tasks:
- id: dbt_build
type: io.kestra.plugin.dbt.cli.DbtCLI
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
containerImage: ghcr.io/kestra-io/dbt-duckdb:latest
commands:
- git clone --depth 1 https://github.com/dbt-labs/jaffle_shop_duckdb.git --branch duckdb --single-branch
- cp -a jaffle_shop_duckdb/. .
- dbt deps
- dbt build
This flexibility allows you to test your tasks locally before deploying them on Kubernetes.
Conclusion
Kestra provides a powerful and flexible way to execute long running and intensive tasks on Kubernetes. By leveraging Kestra’s Task Runners, you can easily define workflows that benefit from Kestra's plugin system while still leveraging the flexibility of Kubernetes to secure and scale the tasks effectively.
If needed, you can also use the podCreate
task to launch a Kubernetes pod directly by providing the complete Kubernetes YAML configuration as input.
Was this page helpful?