Self-Healing Recovery Loop with Retries, Rollback, and an SLA Gate

Most orchestration failures do not need a human; they need a policy. This Kestra blueprint packs four production resilience patterns into a single runnable flow: automatic retries that absorb a flaky API, error-branch rollback when a deploy's smoke test fails, a self-healing loop that restarts an unhealthy dependency and polls until it recovers, and a flow-level SLA that cancels a runaway execution and hands off to an automated root-cause-analysis flow.

How it works

1. The automatic_restart_flaky_api task (io.kestra.plugin.core.http.Request) calls https://httpbin.org/status/200,500, which returns 200 or 500 at random. An exponential retry policy (maxAttempts: 5, delayFactor: 2) restarts the task until it succeeds.
2. The deploy_with_rollback group (io.kestra.plugin.core.flow.Sequential) sets an app_version KV entry to v2, simulates a deploy, then fails its smoke test on purpose (io.kestra.plugin.core.execution.Fail). Its errors branch rolls the KV entry back to v1. allowFailure: true keeps this handled failure from ending the flow.
3. seed_flaky_system_down (io.kestra.plugin.core.kv.Set) marks a simulated system unhealthy, then recovery_loop (io.kestra.plugin.core.flow.LoopUntil) repeatedly calls the flaky-system-health-check subflow. When the probe reports failure, the loop calls the reset-flaky-system recovery subflow and tries again, up to 5 iterations. Because LoopUntil conditions only see outputs of direct children, the probe lives in a subflow and reports back via {{ outputs.attempt_flaky_call.outputs.status }}.
4. guaranteed_failure_sla (io.kestra.plugin.scripts.shell.Commands) sleeps 60 seconds, deliberately pushing total runtime past the flow's MAX_DURATION SLA of PT90S. The SLA cancels the execution and stamps it with the sla: breach label, which is exactly what the companion RCA flow's io.kestra.plugin.core.trigger.Flow precondition listens for.

What you get

• Retries, rollback, self-healing, and SLA enforcement expressed declaratively, no wrapper scripts.
• A recovery loop whose every attempt, probe, and repair is a logged execution you can replay.
• A cancelled-by-SLA execution that automatically produces an RCA report via the linked flow.
• A live-fix story: change sleep 60 to sleep 5, save, and replay the failed task; Kestra reuses all prior task outputs.

Who it's for

• SRE and platform engineers encoding runbooks (restart, poll, escalate) as orchestration.
• Data platform teams who want pipelines to heal themselves before paging anyone.
• Solutions engineers demonstrating failure handling end to end without real infrastructure.

Why orchestrate this with Kestra

Cron and CI runners treat failure as terminal: something exits non-zero, the job goes red, a human digs through logs. Kestra treats failure as a first-class state you can route. Retries are a task property, rollback is an errors branch, self-healing is LoopUntil plus subflows, and SLAs are flow-level policy that can cancel and label an execution so downstream automation reacts to it. Every attempt is preserved with logs and outputs, and task-level replay means a fix reruns only what failed.

Prerequisites

• Kestra 0.18+ (flow-level sla support) with outbound HTTPS access to httpbin.org.
• The two subflows deployed in the same namespace: flaky-system-health-check and reset-flaky-system (see Links).
• Optionally the sla-breach-root-cause-analysis flow deployed so the SLA breach triggers an RCA automatically.

Secrets

No secrets are required; every dependency is simulated with httpbin and the KV store. When you point the tasks at real systems, store credentials as secrets and reference them with {{ secret('NAME') }}.

Quick start

1. Import the two subflows and the RCA flow from the Links section, then import this flow, all into company.team.
2. Execute the flow and watch the Gantt view: retries on task 1, the rollback branch on task 2, and red-then-green iterations inside recovery_loop.
3. After roughly 90 seconds the SLA cancels the execution; open the RCA flow's executions to see the automatic report.
4. Change sleep 60 to sleep 5 in guaranteed_failure_sla, save, and replay that task to finish green.

How to extend

• Point automatic_restart_flaky_api at a real API and tune the retry policy.
• Replace the simulated deploy with real tasks (Terraform, kubectl, dbt) and keep the same errors rollback shape.
• Swap the KV reset in the recovery subflow for a real restart action.
• Change the SLA behavior to FAIL or add a second, shorter SLA with NONE to only label slow runs.
• Add io.kestra.plugin.notifications.slack.SlackIncomingWebhook to the errors branch for on-call visibility.

Links

• Non-failing health check subflow: https://kestra.io/blueprints/flaky-system-health-check
• Recovery subflow that heals the system: https://kestra.io/blueprints/reset-flaky-system
• Automated RCA on SLA breach: https://kestra.io/blueprints/sla-breach-root-cause-analysis