Tasks are discrete actions within a flow, capable of taking inputs and variables from the flow, and producing outputs for downstream consumption by end users and other tasks.
Kestra orchestrates your flows using Flowable Tasks. These tasks do not perform any heavy computation. Instead, they control the orchestration behavior, allowing you to implement more advanced workflow patterns.
In Kestra, most data processing workloads are executed using Runnable Tasks.
In contrast to Flowable Tasks, Runnable Tasks are responsible for performing the actual work. For example, file system operations, API calls, database queries, etc. These tasks can be compute-intensive and are handled by workers.
The table below lists the core task properties available to all tasks.
|The flow identifier, must be unique inside a flow.
|The Java Fully Qualified Class Name of the task.
|The description of the task
|Task retry behavior
|Task timeout expressed in ISO 8601 Durations.
|Set it to
true to disable execution of the task.
|To execute this task on a specific Worker Group (EE).
|To define the log level granularity for which logs will be inserted into the backend database. By default, all logs are stored. However, if you restrict that to e.g., the
INFO level, all lower log levels such as
DEBUG and TRACE won't be persisted.
|Boolean flag allowing to continue the execution even if this task fails.
Task properties can be static or dynamic. Dynamic task properties can be set using epressions. To find out whether a given task property is static or dynamic, check the task documentation available on the plugin's homepage as well as in the UI when you hover over a task and click on the documentation tab on the right.
Often some task properties are marked as not dynamic because they are of complex types (e.g. maps, list of strings, list of maps), meaning that they are placeholders for other dynamic properties. Let's take the runTasks property of Databrick's
SubmitRun task as an example. This property is not dynamic because it's an array of RunSubmitTaskSetting. This
RunSubmitTaskSetting in turn is a group of multiple other properties which are also either dynamic or of complex type (placeholder for other properties). It's therefore useful to always drill down to the lowest level — most lowest level properties are dynamic and can be templated using epressions.