Automated Root Cause Analysis on SLA Breach

When an execution blows its SLA, the slow part is rarely the fix; it is the triage. This Kestra blueprint automates that triage: a Flow trigger watches a specific flow for executions that ended CANCELLED with an sla: breach label, and the instant one appears it produces a root-cause report naming the offending task, the reason, and the exact replay procedure. Failure analysis becomes an orchestrated reaction instead of a Slack thread.

How it works

1. The on_recovery_loop_sla_breach trigger (io.kestra.plugin.core.trigger.Flow) uses preconditions to match executions of company.team.self-healing-recovery-loop that reached the CANCELLED state and carry the label sla: breach. That label is stamped by the source flow's own MAX_DURATION SLA definition, so only genuine SLA cancellations fire the trigger, not manual kills.
2. The log_rca task (io.kestra.plugin.core.log.Log) renders the report using trigger context: {{ trigger.namespace }}, {{ trigger.flowId }}, {{ trigger.executionId }}, and {{ trigger.state }}. In this demo pairing, the root cause is known by construction (the upstream guaranteed_failure_sla task sleeps 60 seconds against a 90-second SLA), so the report can point at the exact line and prescribe the fix: change sleep 60 to sleep 5 and replay the failed task.

What you get

• Event-driven RCA that starts the moment the SLA cancels the upstream execution, with zero polling.
• Precise targeting: state plus label preconditions mean no false positives from unrelated failures or manual cancellations.
• Full lineage from breach to report; the RCA execution records which execution id triggered it.
• A replay-oriented runbook: fix one task, replay from it, and reuse every earlier task's outputs.

Who it's for

• SRE and on-call engineers who want first-line triage done before they open the incident.
• Platform teams standardizing how SLA breaches are documented across hundreds of flows.
• Solutions engineers demonstrating Kestra's event-driven failure handling end to end.

Why orchestrate this with Kestra

Most schedulers can alert that something failed; very few can orchestrate what happens next. Kestra's Flow trigger turns execution state into an event source, and label preconditions let SLA policy (defined in the source flow) drive downstream automation without any glue code. Because the RCA is itself a flow, it can grow from a log line into a pipeline: query the failed execution via the API, attach logs, open a ticket, and notify the owning team, all version-controlled in YAML.

Prerequisites

• Kestra 0.18+ with the companion flow self-healing-recovery-loop deployed in company.team (see Links), including its sla block that applies the sla: breach label with behavior: CANCEL.
• Both flows in the same tenant so the Flow trigger can observe the source executions.

Secrets

No secrets are required for the logging version. If you extend the report to notify or file tickets, store credentials such as SLACK_WEBHOOK_URL or JIRA_API_TOKEN as secrets and reference them with {{ secret('NAME') }}.

Quick start

1. Import this flow and the companion recovery-loop flow into company.team.
2. Execute self-healing-recovery-loop; after about 90 seconds its SLA cancels the run and labels it sla: breach.
3. Watch this flow start automatically and open the log_rca output to read the report.
4. Follow the report's steps: fix the sleep, save, and replay the failed task in the source execution.

How to extend

• Replace the static root-cause text with dynamic analysis: call the Kestra API to pull the breached execution's task durations and logs, or feed them to an LLM task for a generated summary.
• Swap or augment Log with io.kestra.plugin.notifications.slack.SlackExecution or a PagerDuty/Opsgenie task for real alerting.
• File a ticket automatically with a Jira or ServiceNow task, embedding {{ trigger.executionId }} for traceability.
• Broaden preconditions to a whole namespace of flows and route on {{ trigger.flowId }} inside the report.

Links

• Self-healing recovery loop (the monitored flow): https://kestra.io/blueprints/self-healing-recovery-loop
• Non-failing health check subflow: https://kestra.io/blueprints/flaky-system-health-check
• Recovery subflow that heals the system: https://kestra.io/blueprints/reset-flaky-system