Errors and retries
Errors handling
Errors are special branches of your flow where you can define how to handle task failures.
Two kinds of error handlers can be defined:
- Global: error handling global to a flow that must be at the root of the flow.
- Local: error handling local to a Flowable Task, will handle errors for the flowable task and its children.
Global Error Handler
This flow example has a single Bash task that fails immediately.
The global error handler will then be called so the 2nd
task will run.
id: errors
namespace: io.kestra.tests
tasks:
- id: failed
type: io.kestra.core.tasks.scripts.Bash
commands:
- exit 1
errors:
- id: 2nd
type: io.kestra.core.tasks.log.Log
message: I'm failing {{task.id}}
level: INFO
Local Error Handler
In this flow example, the error branch will be used only if a child of the task t2
has an error. If the task t1
failed, the error branch would not be used.
This can be useful to restrict error handling for a specific part of the flow and perform specific tasks like resource cleanup.
id: errors
namespace: io.kestra.tests
tasks:
- id: parent-seq
type: io.kestra.core.tasks.flows.Sequential
tasks:
- id: t1
type: io.kestra.core.tasks.debugs.Return
format: "{{task.id}} > {{taskrun.startDate}}"
- id: t2
type: io.kestra.core.tasks.flows.Sequential
tasks:
- id: t2-t1
type: io.kestra.core.tasks.scripts.Bash
commands:
- 'exit 1'
errors:
- id: error-t1
type: io.kestra.core.tasks.debugs.Return
format: "Error Trigger ! {{task.id}}"
Retries
Kestra provides a task retry feature. This makes it possible to add retry behavior for any task failed run based on configurations in the flow description.
A retry on a task run will create a new task attempt.
Example
The following example defines a retry for the retry-sample
task with a maximum of 5 attempts every 15 minutes:
- id: retry-sample
type: io.kestra.core.tasks.log.Log
message: my output for task {{task.id}}
timeout: PT10M
retry:
maxAttempt: 5
type: constant
interval: PT15M
Retry options for all retry types
name | type | description |
---|---|---|
type | string | Retry behavior to apply. Can be one of constant , exponential , random . |
maxAttempt | integer | Number of retries performed before the system stops retrying. |
maxDuration | Duration | Maximum delay the execution is retried. Once passed, the task is no more processed. |
warningOnRetry | Boolean | Flag the execution as warning if any retry was done on this task. Default false . |
Duration
Some options above have to be filled with a duration notation. Durations are expressed in ISO 8601 Durations, here are some examples:
name | description |
---|---|
PT0.250S | 250 milliseconds delay |
PT2S | 2 seconds delay |
PT1M | 1 minute delay |
PT3.5H | 3 hours and a half delay |
Retry types
constant
This establishes constant retry times: if the interval
property is set to 10 minutes, it retries every 10 minutes.
name | type | description |
---|---|---|
interval | Duration | Duration between each retry. |
exponential
This establishes retry behavior that waits longer between each retry e.g. 1s, 5s, 15s, ...
name | type | description |
---|---|---|
interval | Duration | Duration between each retry. |
maxInterval | Duration | Max Duration between each retry. |
delayFactor | Double | Multiplier for the interval on between retry, default is 2. For example, with an interval of 30s and a delay factor of 2, retry will append at 30s, 1m30, 3m30, ... |
random
This establishes retries with a random delay within minimum and maximum limits.
name | type | description |
---|---|---|
minInterval | Duration | Minimal duration between each retry. |
maxInterval | Duration | Maximum duration between each retry. |
Retry vs. Restart vs. Replay
Automatic vs. manual
Retries ensure that failed task runs are automatically rerun within the same Execution. Apart from retries, defined within your flow code, you can also manually rerun the flow from the Flow Execution page in the UI using the Restart or Replay buttons.
Restart vs. Replay
While Restart will rerun failed tasks within the current Execution (i.e., without creating a new execution), a Replay would result in a completely new run with a different Execution ID than the initial run.
When you replay an Execution, a new execution gets created for the same flow. However, you can still track which Execution triggered this new run thanks to the Original Execution
field:
Replay can be executed from any task, even if that task executed successfully. But note that when you trigger a replay from a specific failed task, it will still result in a new Execution running all tasks downstream of your chosen start task:
When you want to rerun only failed tasks, use Restart
.
Summary: Retries vs. Restart vs. Replay
The table below summarizes the differences between a retry, restart and replay.
Concept | Flow or task level | Automatic or manual | Does it create a new execution? |
---|---|---|---|
Retry | Task level | Automatic | No, it only reruns a given task within the same Execution. Each retry results in a new attempt number, allowing you to see how many times a given task run was retried. |
Restart | Flow level | Manual | No, it only reruns all failed tasks within the same Execution. It's meant to handle unanticipated, transient failures. The UI shows a new attempt number for all task runs that were restarted. |
Replay | Either flow or task level | Manual | Yes. You can pick an arbitrary step from which a new execution should be triggered. If you select a task in the middle that needs outputs from a previous task, its output is retrieved from cache. |