​Developer ​Guide

Data storage and processing

Kestra's primary purpose is to orchestrate data processing via tasks, so data is central to each flow's execution.

Errors and retries

Errors are special branches of a flow where you can define how to react to task failures (e.g. send an email, open a ticket, etc.).

Namespace Files

This page covers the concept of Namespace Files and how to use them in your flows.

Version Control with Git

Kestra supports version control with Git. You can use one or more Git repositories to store your Flows and Namespace Files, and track changes to them over time via Git commit history.

Caching

Kestra provides a file caching, which is especially useful when you work with sizeable package dependencies that don't change often.

Build a Custom Plugin

Browse Kestra's integrations and learn how to create your own plugins.

Programming Languages

Kestra currently supports the following programming languages: Python, R, Node.js, Shell, Powershell, Julia and Ruby.

Commands and Script tasks

For each of the supported languages (a.o. Python, R, Node.js, Shell), Kestra provides two types of tasks: Script and Commands.

Inline Scripts in Docker

To get started with a Script task, paste your custom script inline in your YAML workflow definition along with any other configuration.

DOCKER and PROCESS runners

Kestra supports two runners for scripting tasks: DOCKER and PROCESS.

Building a custom Docker image

This page shows how to build a custom Docker image for your script tasks.

Installing dependencies at runtime

There are several ways of installing custom packages for your workflows. This page shows how to install dependencies at runtime using the beforeCommands property.

Additional languages (Rust)

Kestra is language agnostic. This section shows how to use Rust in your flows.

Bind mount

To run a script stored locally, you can bind-mount it to your Kestra container.

The Git Clone task

This task allows you to clone a Git repository into a working directory and then use the files from that repository in downstream tasks.

Python, R, Node.js and Shell Scripts

Kestra is language agnostic. You can orchestrate custom business logic written in Python, R, Julia, Ruby, Node.js, Powershell and Shell scripts, and you can also build custom plugins in Java.

Output directory

If you want to generate files in your script to make them available for download and use in downstream tasks, you can leverage either the outputFiles property or the outputDir expression.

Outputs and metrics

Your scripts can send outputs and metrics to Kestra's backend during flow execution. This allows you to track custom metadata and visualize it across multiple executions of a flow.

Working Directory

This task allows you to run multiple tasks sequentially in the same working directory. It is useful when you want to share files from Namespace Files or from a Git repository across multiple tasks.

GitHub Actions

We provide two official GitHub Actions to help you create a CI/CD pipeline for your Kestra flows.

GitLab CI

This page describes how to use GitLab CI to create a CI/CD pipeline for your Kestra flows.

Terraform

This page describes how to use Terraform to provision and manage changes to Kestra resources.

Helpers

Kestra provides some helper functions that can help during flow development.

Azure DevOps

This page describes how to use Azure DevOps to create a CI/CD pipeline for your Kestra flows.

BitBucket Pipes

This page describes how to use BitBucket Pipes to create a CI/CD pipeline for your Kestra flows.

Create a CI/CD pipeline

Kestra provides several ways to create a CI/CD pipeline for your flows. This section explains how to automate the validation and deployment of your workflows using CI/CD.