​Flow

Flow is a container for tasks and their orchestration logic.

Components of a flow

A flow is a container for tasks, their inputs, outputs, handling of errors and overall orchestration logic. It defines the order in which tasks are executed and how they are executed, e.g. sequentially, in parallel, based on upstream task dependencies and their state, etc.

You can define a flow declaratively using a YAML file.

A flow must have:

• identifier (id)
• namespace
• list of tasks

Optionally, a flow can also have:

• inputs
• outputs
• variables
• triggers
• labels
• pluginDefaults
• errors
• retries
• timeout
• concurrency
• descriptions
• disabled
• revision

Flow sample

Here is a sample flow definition. It uses tasks available in Kestra core for testing purposes, such as the Return or Log tasks, and demonstrates how to use labels, inputs, variables, triggers and various descriptions.

id: hello-world
namespace: company.team

description: flow **documentation** in *Markdown*

labels:
  env: prod
  team: engineering

inputs:
  - id: my-value
    type: STRING
    required: false
    defaults: "default value"
    description: This is a not required my-value

variables:
  first: "1"
  second: "{{vars.first}} > 2"

tasks:
  - id: date
    type: io.kestra.plugin.core.debug.Return
    description: "Some tasks **documentation** in *Markdown*"
    format: "A log line content with a contextual date variable {{taskrun.startDate}}"

pluginDefaults:
  - type: io.kestra.plugin.core.log.Log
    values:
      level: ERROR

Plugin defaults

You can also define pluginDefaults in your flow. This is a list of default task properties that will be applied to each task of a certain type inside your flow. The pluginDefaults property can be handy to avoid repeating the same values when leveraging the same task multiple times.

Variables

You can set flow variables that will be accessible by each task using {{ vars.key }}. Flow variables is a map of key/value pairs.

List of tasks

The most important part of a flow is the list of tasks that will be run sequentially when the flow is executed.

Disable a flow

By default, all flows are active and will execute whether or not a trigger has been set.

You have the option to disable a Flow, which is particularly useful when you want to temporarily stop a Flow from executing e.g. when troubleshooting a failure.

Task

A task is a single action in a flow. A task can have properties, use flow inputs and other task's outputs, perform an action, and produce an output.

There are two kinds of tasks in Kestra:

• Runnable Tasks
• Flowable Tasks

Runnable Task

Runnable Tasks handle computational work in the flow. For example, file system operations, API calls, database queries, etc. These tasks can be compute-intensive and are handled by workers.

By default, Kestra only includes a few Runnable Tasks. However, many of them are available as plugins, and if you use our default Docker image, plenty of them will already be included.

Flowable Task

Flowable Tasks only handle flow logic (branching, grouping, parallel processing, etc.) and start new tasks. For example, the Switch task decides the next task to run based on some inputs.

A Flowable Task is handled by an executor and can be called very often. Because of that, these tasks cannot include intensive computations, unlike Runnable Tasks. Most of the common Flowable Tasks are available in the default Kestra installation.

Labels

Labels are key-value pairs that you can add to flows. Labels are used to organize flows and can be used to filter executions of any given flow from the UI.

Inputs

Inputs are parameters sent to a flow at execution time. It's important to note that inputs in Kestra are strongly typed.

The inputs can be declared as either optional or mandatory. If the flow has required inputs, you'll have to provide them before the execution of the flow. You can also provide default values to the inputs.

Inputs can have validation rules that are enforced at execution time.

Inputs of type FILE will be uploaded to Kestra's internal storage and made available for all tasks.

Flow inputs can be seen in the Overview tab of the Execution page.

Outputs

Each task (or flow) can produce outputs that may contain multiple properties. This output is described in the plugin documentation task and can then be accessible by all following tasks via expressions.

Some outputs are of a special type and will be stored in Kestra's internal storage. Kestra will automatically make these outputs available for all tasks.

You can view:

• task outputs in the Outputs tab of the Execution page.
• flow outputs in the Overview tab of the Execution page.

If an output is a file from the internal storage, it will be available to download.

For more details on both task and flow outputs, see the Outputs page.

Revision

Changing the source of a flow will produce a new revision for the flow. The revision is an incremental number that will be updated each time you change the flow.

Internally, Kestra will track and manage all the revisions of the flow. Think of it as version control for your flows integrated inside Kestra.

You can access old revisions inside the Revisions tab of the Flows page.

Triggers

Triggers are a way to start a flow from external events. For example, a trigger might initiate a flow at a scheduled time or based on external events (webhooks, file creation, message in a broker, etc.).

Flow variable expressions

Flows have a number of variable expressions giving you information about them dynamically, a few examples include:

Listeners (deprecated)

Listeners are special tasks that can listen to the current flow, and launch tasks outside the flow, meaning launch tasks that are not part of the flow.

The result of listeners will not change the execution status of the flow. Listeners are mainly used to send notifications or handle special behavior outside the primary flow.

Templates (deprecated)

Templates are lists of tasks that can be shared between flows. You can define a template and call it from other flows. Templates allow you to share a list of tasks and keep them updated without changing all flows that use them.

FAQ

Where does Kestra store flows?

Flows are stored in a serialized format directly in the Kestra backend database.

The easiest way to add new flows is to add them directly from the Kestra UI. You can also use the Git Sync pattern or CI/CD integration to add flows automatically after a pull request is merged to a given Git branch.

To see how flows are represented in a file structure, you can leverage the _flows directory in the Namespace Files editor.

How to load flows at server startup?

If you want to load a given local directory of flows to be loaded into Kestra (e.g. during local development), you can use the -f or --flow-path flag when starting Kestra:

./kestra server standalone -f /path/to/flows

That path should point to a directory containing YAML files with the flow definition. These files will be loaded to the Kestra repository at startup. Kestra will make sure to add flows to the right namespace, as declared in the flow YAML definition.

For more information about the Kestra server CLI, check the Server CLI Reference section.

Can I sync a local flows directory to be continuously loaded into Kestra?

At the time of writing, there is no syncing of a flows directory to Kestra. However, we are aware of that need and we are working on a solution. You can follow up in this GitHub issue.