Layered Error Handling with Retries and KV Context

Learn every layer of Kestra's failure handling model with one deliberately unreliable flow. A Python task raises an exception 99 percent of the time, which lets you watch, in order: the flow level retry policy re-running the failed task, the errors block inside the Sequential group firing with block scoped context, the flow level errors block firing with {{ errorLogs() }} output, and a KV store write that leaves behind context an external alerting flow can pick up. It is the reference companion to a namespace scoped alerting flow and a resubmission flow, together forming a complete detect, notify, retry pattern.

How it works

1. The task_block group (io.kestra.plugin.core.flow.Sequential) contains log_message (io.kestra.plugin.core.log.Log) and fail_task (io.kestra.plugin.scripts.python.Script), which raises a RuntimeError when random.random() <= 0.99.
2. The flow level retry policy (behavior: RETRY_FAILED_TASK, type: constant, maxAttempt: 2, interval: PT1S) retries only the failed task before the failure propagates.
3. When retries are exhausted, the errors block attached to the Sequential group runs first: log_error_seq_block logs the failing {{ task.id }}, the {{ taskrun.id }} with its start date, and the collected {{ errorLogs() }}.
4. The flow level errors block then runs: log_error_flow records the same debugging context at flow scope, and set_custom_vars (io.kestra.plugin.core.kv.Set) writes <flowId>-custom-label = marketing-report to the KV store so alerting flows can enrich their notifications.
5. The webhook_call trigger (io.kestra.plugin.core.trigger.Webhook) lets any external system, or a quick curl, start the flow on demand with a secret key.

What you get

• A safe sandbox to observe retries, block errors, and flow errors firing in their real order.
• Ready to copy errors blocks with the exact template expressions for task, taskrun, and error logs.
• A KV store handoff pattern that decouples failing pipelines from alerting flows.
• A webhook entry point for repeatable failure drills from CI or the command line.
• Roughly one run in a hundred that succeeds, proving the happy path works too.

Who it's for

• Engineers new to Kestra who want to understand error handling scopes hands on.
• Platform teams designing standard failure handling conventions for their organization.
• On call engineers building and testing alerting flows that need a reliable failure generator.

Why orchestrate this with Kestra

Most orchestrators give you a single on failure callback. Kestra layers the model: retries happen first and are policy, not code; errors blocks can be attached to any task group for local cleanup; a flow level errors block catches anything that escapes; and functions like errorLogs() expose the actual failure text to your handlers. Because the KV store persists context beyond the execution, downstream alerting and resubmission flows can react with full knowledge of what broke.

Prerequisites

• A Kestra instance with the Python script plugin available (the default image works).
• Optionally, a companion alerting flow with a Flow trigger to observe the failures externally.

Secrets

• ERROR_HANDLING_WEBHOOK_KEY: shared secret guarding the Webhook trigger so only authorized callers can start failure drills.

Quick start

1. Add the ERROR_HANDLING_WEBHOOK_KEY secret to your Kestra namespace.
2. Deploy the flow and execute it manually, or POST to the webhook URL with the key.
3. Watch the Gantt view: the retry attempt, then the block error handler, then the flow error handlers.
4. Open the KV store and confirm the <flowId>-custom-label entry was written.

How to extend

• Lower the failure probability to see mixed success and failure behavior across runs.
• Switch the retry policy to type: exponential with a maxDuration for production realism.
• Add io.kestra.plugin.slack.notifications.SlackIncomingWebhook to the flow errors block for direct alerts.
• Pair with a Flow triggered alerting blueprint that reads the KV label for enriched notifications.
• Replace the random failure with a real flaky dependency, such as an HTTP call to a staging service.

Links

• Error handling docs: https://kestra.io/docs/workflow-components/errors
• Retries docs: https://kestra.io/docs/workflow-components/retries
• KV store docs: https://kestra.io/docs/concepts/kv-store