Kubernetes Deployment Restart Remediation from Alerts

Turn "have you tried restarting it" into a governed automation. When a monitoring alert fires, it posts the namespace and deployment name to a secured webhook; the flow validates both against an allowlist, performs a rolling restart with kubectl, then polls the deployment status with exponential backoff until the rollout is healthy. Success and failure both land in Slack, so on-call engineers see remediation happen without touching a terminal.

How it works

1. The alert_webhook trigger (io.kestra.plugin.core.trigger.Webhook) accepts POSTs secured by K8S_REMEDIATION_WEBHOOK_KEY; Alertmanager, Datadog, or any monitor can call it with namespace and deployment_name query parameters.
2. The parse_webhook_payload task (io.kestra.plugin.core.execution.SetVariables) resolves the target from trigger.parameters with ?? fallbacks to flow variables.
3. The validate_namespace task (io.kestra.plugin.core.flow.If) fails fast when the deployment name is missing or the namespace is not in allowed_namespaces, preventing alerts from restarting arbitrary workloads.
4. The restart_service task (io.kestra.plugin.kubernetes.kubectl.Restart) performs the rolling restart of the Deployment.
5. The get_deployment_status task (io.kestra.plugin.kubernetes.kubectl.Get with fetchType: FETCH_ONE) fetches the deployment and retries with an exponential policy (5 attempts, PT15S initial interval, PT5M cap) until the status reflects a healthy rollout.
6. notify_slack_success posts the verified status; a flow-level errors handler posts a failure message if validation, restart, or verification fail. Cluster credentials are injected once via pluginDefaults for all io.kestra.plugin.kubernetes tasks.

What you get

• Self-healing for the most common K8s incident class, with humans informed rather than involved.
• A namespace allowlist that bounds the blast radius of automated restarts.
• Verified remediation: the flow only succeeds when the rollout is actually healthy.
• Cluster credentials from secrets, applied through a single pluginDefaults block.

Who it's for

• SRE teams automating runbook step one for crashlooping or degraded deployments.
• Platform teams offering guarded self-service restarts to application teams.
• On-call engineers who want alerts to arrive already remediated.

Why orchestrate this with Kestra

A webhook receiver script with kubectl access is a security incident waiting to happen. Kestra adds the guardrails: a keyed webhook, explicit allowlist validation before any cluster call, secrets management for the cluster token, exponential-backoff verification as a declarative retry policy rather than a sleep loop, and an execution log that shows exactly which alert restarted what and whether it recovered.

Prerequisites

• A Kubernetes cluster reachable from the Kestra worker.
• A service account token with permission to restart and get deployments in the allowed namespaces.
• A monitoring system able to POST to the webhook with the target parameters.

Secrets

• K8S_REMEDIATION_WEBHOOK_KEY: shared key guarding the webhook trigger.
• K8S_MASTER_URL, K8S_CA_CERT_DATA, K8S_OAUTH_TOKEN: cluster connection settings used by all Kubernetes tasks.
• SLACK_WEBHOOK_URL: Slack incoming webhook for remediation reports.

Quick start

1. Add the secrets above to your Kestra namespace.
2. Adjust allowed_namespaces to the namespaces you are willing to auto-remediate.
3. Test with a manual POST to the webhook: ?namespace=staging&deployment_name=my-app.
4. Point your alerting rule at the webhook URL.

How to extend

• Add a io.kestra.plugin.core.flow.Pause approval for production namespaces while auto-remediating staging.
• Capture pod logs before restarting with io.kestra.plugin.kubernetes.kubectl.Get on pods for post-incident analysis.
• Track remediation counts per deployment in the KV store and open a ticket after N restarts in 24 hours.
• Fan out to multiple clusters by parameterizing the connection via inputs.

Links

• Kubernetes plugin docs: https://kestra.io/plugins/plugin-kubernetes
• Webhook trigger docs: https://kestra.io/plugins/core/triggers/io.kestra.plugin.core.trigger.webhook
• Kubernetes orchestration with Kestra: https://kestra.io/orchestration/kubernetes