Helm Release Failure Auto-Rollback Sentinel

A Helm release can fail from a CI pipeline, a GitOps controller, or someone running helm directly on a laptop, and none of those paths necessarily tell anyone when it happens. helm status can be polled on a schedule to check one release you already know to watch, but it says nothing about a release nobody is specifically looking for right now. This blueprint watches the whole cluster's release state directly through the Kubernetes API, tells a first failure apart from a release that has now failed twice in a row, and only rolls the repeat offender back once you have explicitly said this flow may do that.

This blueprint was created by Mohit Jeswani.

How it works

1. on_release_failed (io.kestra.plugin.helm.ReleaseTrigger) polls Helm's release Secrets cluster-wide and fires whenever a release's latest revision reaches FAILED, carrying each release's name, namespace, revision, and previousStatus in trigger.releases.
2. evaluate (io.kestra.plugin.core.output.OutputValues) splits the fire into first-time failures and releases whose previousStatus was already failed, then classifies the whole fire as FIRST_FAILURE_ALERT, REPEATED_FAILURE_NEEDS_HUMAN, or REPEATED_FAILURE_AUTO_ROLLBACK depending on repeated_failure_action.
3. route (io.kestra.plugin.core.flow.Switch):
   • FIRST_FAILURE_ALERT / REPEATED_FAILURE_NEEDS_HUMAN: posts the release summary to Slack.
   • REPEATED_FAILURE_AUTO_ROLLBACK: Loops over the fired releases, rolls back only the ones whose previousStatus was already failed with io.kestra.plugin.helm.Rollback, then posts a Slack notice that it happened automatically.
4. summary logs the count and severity, closing the governance cycle.
5. errors alerts Slack if the flow itself fails, for example a revoked cluster credential.

What you get

• A failure caught however it happened on the cluster, not only one your own Kestra flow caused, since the trigger reads Helm's release state directly rather than wrapping one specific deploy step.
• A rollback that only ever fires on a release that has now failed twice in a row, so a one-off failure gets a human's attention first instead of being silently reverted.
• A read-only path by default: nothing is rolled back unless repeated_failure_action is explicitly set to AUTO_ROLLBACK.

Who it's for

• Platform and SRE teams running Helm across a cluster who want release failures caught no matter which pipeline or person triggered them.
• Teams evaluating the Helm plugin who want a worked example of ReleaseTrigger used as a standing, cluster-wide sentinel, distinct from this repo's existing Helm blueprints: one polls helm status for one named release on a schedule, the other rolls back inside a single CI/CD upgrade step, while this one is event-driven, cluster-wide, and reacts to any failure however it originated.

Why orchestrate this with Kestra

Helm's release Secrets record what happened, but nothing in Helm itself watches them, tells a first failure apart from a repeat, or pages a channel. Kestra turns ReleaseTrigger's poll into a severity decision, keeps repeated_failure_action as an explicit flow input instead of a hardcoded always-rollback script, and keeps a full execution history of every failure seen and what this flow did about it.

Prerequisites

• A Kubernetes cluster with Helm releases installed using Helm's default secret storage driver, and RBAC granting list/get on secrets in the watched namespaces.
• A release you can force to fail twice in a row (for example, upgrading to a chart version with a bad values file) to exercise both the alert and the rollback path.
• A Slack incoming webhook URL.

A local cluster via kind (Kubernetes-in-Docker, a standard open-source Docker-based tool) is enough to install a chart, fail an upgrade on purpose, and watch this sentinel react.

Secrets

• SLACK_WEBHOOK_URL: used by all three Slack payloads and the errors block. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_SLACK_WEBHOOK_URL.
• KUBE_MASTER_URL: the cluster's API server URL, used by both connection blocks. When the Kestra worker runs as a pod inside the target cluster, drop both connection blocks entirely instead; the plugin then picks up the in-cluster service account on its own.
• KUBE_OAUTH_TOKEN: a bearer token for a service account with permission to read release Secrets (for the trigger) and run upgrades/rollbacks (for rollback). Generate one with kubectl create token <service-account> -n <namespace>.

Inputs

Input	Default	Purpose
repeated_failure_action	ALERT_ONLY	Set to AUTO_ROLLBACK to let this flow roll back a release that has failed twice in a row

Outputs

• {{ trigger.releases }} / .count: the releases that reached FAILED this fire, and how many, as ReleaseTrigger reports them.
• {{ outputs.evaluate.values.failure_count }} / .repeated_failure_count / .release_summary: the total and repeated-failure counts, and a readable summary of every failed release.
• {{ outputs.evaluate.values.severity }}: FIRST_FAILURE_ALERT, REPEATED_FAILURE_NEEDS_HUMAN, or REPEATED_FAILURE_AUTO_ROLLBACK.

Pitfalls

• connection, allNamespaces, and statuses are trigger-level configuration, not flow inputs; a polling trigger has no per-execution input overrides the way a Schedule trigger does, so watching a different scope means editing the trigger block directly.
• previousStatus is null the first time ReleaseTrigger ever sees a release, so a release already FAILED when this trigger starts watching it is always classified as a first-time failure on that first fire, even if it had actually failed before the trigger existed.
• Rollback defaults to the revision immediately before the current one; if that revision was itself broken, the auto-rollback path can roll a release back into a different failure rather than a known-good state. Set revision explicitly on rollback for a release where that matters.
• repeated_failure_action applies to every release in every fire, not per release; a cluster-wide AUTO_ROLLBACK setting will roll back every repeated failure this trigger sees, so scope releaseNamePattern or namespace on the trigger to the releases you are actually ready to auto-remediate.

Quick start

1. Add the SLACK_WEBHOOK_URL, KUBE_MASTER_URL, and KUBE_OAUTH_TOKEN secrets. Against a local kind cluster, get the master URL from kubectl cluster-info and a token from kubectl create token <service-account> -n <namespace>. Running the Kestra worker as a pod inside the cluster instead, drop both connection blocks so the plugin picks up the in-cluster service account automatically.
2. Install a chart on that cluster.
3. Enable the on_release_failed trigger with repeated_failure_action: ALERT_ONLY, force an upgrade to fail once, and confirm alert_first_failure fires.
4. Force the same release to fail again and confirm alert_repeated_failure fires instead.
5. Set repeated_failure_action: AUTO_ROLLBACK, force a third failure, and confirm rollback runs and alert_auto_rolled_back fires.

How to extend

• Add stuckFor alongside a second statuses filter on a separate trigger to also catch a release stuck mid-upgrade, not just one that reached a terminal FAILED state.
• Chain a io.kestra.plugin.kubernetes.kubectl.Get lookup of the release's pod status into the Slack alert so a human does not have to go find it manually.
• Scope repeated_failure_action per namespace by running one copy of this flow per environment, AUTO_ROLLBACK in staging and ALERT_ONLY in production.

Links

• Helm plugin: https://kestra.io/plugins/plugin-helm
• Helm release status reference: https://helm.sh/docs/helm/helm_status/
• kind (Kubernetes-in-Docker): https://kind.sigs.k8s.io/
• Slack incoming webhooks: https://api.slack.com/messaging/webhooks