Develop Python, R, Shell, PowerShell, Julia, Ruby or Node.js scripts and integrate them with Git and CI/CD.
Data storage and processing
Kestra's primary purpose is to orchestrate data processing via tasks, so data is central to each flow's execution.
Errors and retries
Errors are special branches of a flow where you can define how to react to task failures (e.g. send an email, open a ticket, etc.).
This page covers the concept of Namespace Files and how to use them in your flows.
Version Control with Git
Kestra supports version control with Git. You can use one or more Git repositories to store your Flows and Namespace Files, and track changes to them over time via Git commit history.
Kestra provides a file caching, which is especially useful when you work with sizeable package dependencies that don't change often.
Kestra currently supports the following programming languages: Python, R, Node.js, Shell, Powershell, Julia and Ruby.
Commands and Script tasks
For each of the supported languages (a.o. Python, R, Node.js, Shell), Kestra provides two types of tasks: Script and Commands.
Inline Scripts in Docker
To get started with a Script task, paste your custom script inline in your YAML workflow definition along with any other configuration.
DOCKER and PROCESS runners
Kestra supports two runners for scripting tasks: DOCKER and PROCESS.
Building a custom Docker image
This page shows how to build a custom Docker image for your script tasks.
Installing dependencies at runtime
There are several ways of installing custom packages for your workflows. This page shows how to install dependencies at runtime using the beforeCommands property.
Additional languages (Rust)
Kestra is language agnostic. This section shows how to use Rust in your flows.
To run a script stored locally, you can bind-mount it to your Kestra container.
The Git Clone task
This task allows you to clone a Git repository into a working directory and then use the files from that repository in downstream tasks.
Python, R, Node.js and Shell Scripts
Kestra is language agnostic. You can orchestrate custom business logic written in Python, R, Julia, Ruby, Node.js, Powershell and Shell scripts, and you can also build custom plugins in Java.
If you want to generate files in your script to make them available for download and use in downstream tasks, you can leverage either the outputFiles property or the outputDir expression.
Outputs and metrics
Your scripts can send outputs and metrics to Kestra's backend during flow execution. This allows you to track custom metadata and visualize it across multiple executions of a flow.
This task allows you to run multiple tasks sequentially in the same working directory. It is useful when you want to share files from Namespace Files or from a Git repository across multiple tasks.
We provide two official GitHub Actions to help you create a CI/CD pipeline for your Kestra flows.
This page describes how to use GitLab CI to create a CI/CD pipeline for your Kestra flows.
This page describes how to use Terraform to provision and manage changes to Kestra resources.
Kestra provides some helper functions that can help during flow development.
This page describes how to use Azure DevOps to create a CI/CD pipeline for your Kestra flows.
This page describes how to use BitBucket Pipes to create a CI/CD pipeline for your Kestra flows.
Create a CI/CD pipeline
Kestra provides several ways to create a CI/CD pipeline for your flows. This section explains how to automate the validation and deployment of your workflows using CI/CD.