Reset a Flaky System from a Self-Healing Recovery Loop

This is the recovery action of a self-healing orchestration pattern. When a parent loop detects that a dependency is unhealthy, it does not page a human first; it calls this subflow, which performs the corrective action and hands control straight back to the loop for another health check. Here the "restart" is simulated by flipping a KV entry, which keeps the blueprint runnable anywhere, but the structure is exactly what you would use to bounce a real service.

How it works

1. The reset_flaky_system_status task (io.kestra.plugin.core.kv.Set) writes 200 to the flaky_system_status KV entry. The companion health-check flow reads this same entry and calls https://httpbin.org/status/<value>, so after the reset the next probe returns HTTP 200.
2. The log_reset task (io.kestra.plugin.core.log.Log) records that recovery ran, giving the parent execution an auditable trace of every self-healing action.

The parent flow invokes this blueprint through io.kestra.plugin.core.flow.Subflow with wait: true and transmitFailed: false, so the loop pauses until recovery completes and is never poisoned by a recovery hiccup.

What you get

• A minimal, reusable recovery action that any flow can call by id.
• A KV-backed simulation of "restart the service" that runs on any Kestra instance with zero external dependencies.
• An audit line in the logs every time self-healing fires, so you can count recoveries over time.
• Clean separation of concerns: detection lives in the health check, remediation lives here, and policy (how many attempts, how long to wait) lives in the parent loop.

Who it's for

• SRE and platform teams automating runbooks that currently say "if the check fails, restart the thing and check again".
• Solutions engineers demonstrating self-healing orchestration without needing real infrastructure to break.
• Teams migrating bash-based remediation cron jobs into observable, versioned flows.

Why orchestrate this with Kestra

Remediation logic buried in a monitoring tool's webhook or a cron script is invisible until it misfires. As a Kestra subflow, every recovery is a first-class execution with its own logs, duration, and revision history. The KV store gives flows a shared, durable state without standing up a database, and the subflow contract means you can swap the simulated reset for a real restart (Kubernetes rollout, systemd unit, cloud API call) without touching the parent loop at all.

Prerequisites

• A Kestra instance with the KV store available in the company.team namespace (built in, no plugin install needed).
• The companion flows deployed if you want the full self-healing scenario: the parent loop and the health check listed under Links.

Secrets

No secrets are required for the simulated reset. If you replace it with a real restart, store credentials such as a kubeconfig token or cloud API key as secrets and reference them with {{ secret('NAME') }}.

Quick start

1. Import this flow into the company.team namespace.
2. Set the flaky_system_status KV entry to 500 to simulate a broken system.
3. Execute the flow, then confirm in Namespaces, KV Store that the value is back to 200.
4. Deploy the parent loop blueprint and watch it call this flow automatically whenever the health check fails.

How to extend

• Replace the KV write with a real remediation task, for example io.kestra.plugin.kubernetes.kubectl.PodDelete to bounce a pod or an HTTP call to a service's restart endpoint.
• Add a notification task after log_reset so on-call sees that self-healing fired even when no human action was needed.
• Record each recovery in a database or KV counter to alert when the system flaps more than N times per hour.
• Accept an input such as service_name to make one recovery flow serve many systems.

Links

• Self-healing recovery loop (parent flow): https://kestra.io/blueprints/self-healing-recovery-loop
• Non-failing health check subflow: https://kestra.io/blueprints/flaky-system-health-check
• Automated RCA on SLA breach: https://kestra.io/blueprints/sla-breach-root-cause-analysis