DeploymentWatch: Kubernetes stuck rollout watcher blueprint for Kestra

id: deploymentwatch
namespace: company.team
description: Watches Kubernetes Deployments and alerts on Slack when a rollout gets stuck.

labels:
  project: deploymentwatch
  team: platform

inputs:
  - id: namespace
    type: STRING
    displayName: "Namespace"
    description: the namespace whose deployments to watch
    defaults: default
  - id: slack_webhook
    type: STRING
    displayName: "Slack Webhook URL"
    description: Incoming webhook URL of the Slack channel for the alerts

triggers:
  - id: schedule
    type: io.kestra.plugin.core.trigger.Schedule
    description: checks deployment rollouts every morning
    cron: "0 8 * * *"
    timezone: UTC
    inputs:
      namespace: default
      slack_webhook: "{{ secret('SLACK_WEBHOOK') }}"

tasks:
  - id: list_deployments
    type: io.kestra.plugin.scripts.shell.Commands
    description: lists all deployments in the namespace as JSON
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
      user: "0"
    containerImage: bitnami/kubectl
    inputFiles:
      client.crt: "{{ secret('CLIENT_CRT') }}"
      client.key: "{{ secret('CLIENT_KEY') }}"
    commands:
      - kubectl get deployments -n {{ inputs.namespace }} -o json
        --client-certificate=client.crt --client-key=client.key --server={{
        secret('MASTERURL') }} --insecure-skip-tls-verify=true >
        deployments.json
    outputFiles:
      - deployments.json

  - id: find_trouble
    type: io.kestra.plugin.scripts.python.Script
    description: finds deployments with stuck rollouts, drops ones already alerted on
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
      user: "0"
    containerImage: python:3.11
    dependencies:
      - kestra
    script: |
      import json
      from kestra import Kestra

      namespace = """{{ inputs.namespace }}"""

      with open("""{{ outputs.list_deployments.outputFiles['deployments.json'] }}""") as f:
          deployments = json.load(f)['items']

      # names we already alerted on, from the KV store (null on first run -> empty)
      raw_state = """{{ kv('deploymentwatch-state-' ~ inputs.namespace, errorOnMissing=false) }}"""
      already = set(json.loads(raw_state) if raw_state.strip() else [])

      troubled = []
      for dep in deployments:
          name = dep['metadata']['name']
          status = dep.get('status', {})
          reasons = []

          unavailable = status.get('unavailableReplicas', 0) or 0
          if unavailable > 0:
              reasons.append(f"{unavailable} unavailable replica(s)")

          for cond in status.get('conditions', []):
              if cond.get('type') == 'Progressing' and cond.get('reason') == 'ProgressDeadlineExceeded':
                  reasons.append("progress deadline exceeded")

          if reasons:
              troubled.append({'name': name, 'reasons': reasons})

      troubled_names = set(t['name'] for t in troubled)
      new_trouble = [t for t in troubled if t['name'] not in already]
      # the new state is simply who is troubled right now;
      # recovered deployments drop out on their own, so a future stuck rollout alerts again

      lines = [f"*DeploymentWatch - stuck rollouts in '{namespace}'*", ""]
      for t in new_trouble:
          lines.append(f"- *{t['name']}*: " + "; ".join(t['reasons']))
          lines.append(f"  inspect: `kubectl rollout status deployment/{t['name']} -n {namespace}`")
      lines.append("")
      alert_text = "\n".join(lines)

      Kestra.outputs({
          'troubled': new_trouble,
          'count': len(new_trouble),
          'new_state_json': json.dumps(sorted(troubled_names)),
          'alert_text': alert_text,
      })

  - id: maybe_alert
    type: io.kestra.plugin.core.flow.If
    description: only alerts when there is something new to report
    condition: "{{ outputs.find_trouble.vars.count > 0 }}"
    then:
      - id: send_alert
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        description: posts the stuck deployments to Slack
        url: "{{ inputs.slack_webhook }}"
        messageText: "{{ outputs.find_trouble.vars.alert_text }}"

  - id: save_state
    type: io.kestra.plugin.core.kv.Set
    description: remembers who is troubled now, so recovered deployments alert again
      next time
    key: "deploymentwatch-state-{{ inputs.namespace }}"
    value: "{{ outputs.find_trouble.vars.new_state_json }}"
    overwrite: true

How it works

list_deployments (io.kestra.plugin.scripts.shell.Commands) runs kubectl get deployments in the target namespace and saves the full deployment list as JSON. It authenticates with a client certificate and key passed as Kestra secrets, the same pattern the Kubernetes gallery blueprints use.
find_trouble (io.kestra.plugin.scripts.python.Script) reads the remembered state from the KV store, then checks every deployment's status. Trouble is any unavailable replicas, or a Progressing condition with reason ProgressDeadlineExceeded. Anything already alerted on is dropped.
maybe_alert (io.kestra.plugin.core.flow.If) runs only when there is something new to report, and send_alert posts the stuck deployments to Slack with the reasons and a kubectl rollout status command to inspect each one.
save_state (io.kestra.plugin.core.kv.Set) writes back the new list of troubled deployment names as one JSON value. Recovered deployments drop out on their own, so a future stuck rollout alerts again.

What you get

A daily rollout health check for any namespace, no agent to install inside the cluster.
Slack alerts only for new trouble: which deployment, why it is stuck, and the kubectl command to look at it.
No alert spam: the KV-backed memory remembers who was already reported, and forgets deployments that recovered.
Quiet runs when every rollout is progressing fine.

Who it's for

Anyone shipping to Kubernetes who has watched a bad image sit in a rollout for hours because nobody noticed. Works on any cluster you can reach with a client certificate: minikube, kind, managed clusters.

Why orchestrate this with Kestra

A shell script on a cron server can list deployments, but then you need somewhere to keep the "already alerted" state, somewhere to hold the secrets, and a place to see the run history when an alert looks wrong. Kestra gives you all three: the KV store is the memory, secrets stay out of the YAML, and every run keeps its logs and outputs where you can inspect them.

Prerequisites

A Kubernetes cluster reachable from where Kestra runs, with a client certificate and key for authentication.
Kestra with the Docker task runner available (used for the kubectl and Python steps).
A Slack Incoming Webhook URL.

Secrets

Secret	What it is
`CLIENT_CRT`	base64-encoded client certificate for the cluster
`CLIENT_KEY`	base64-encoded client key for the cluster
`MASTERURL`	base64-encoded Kubernetes API server URL, reachable from inside the Kestra container
`SLACK_WEBHOOK`	base64-encoded Slack Incoming Webhook URL

In Kestra OSS, secrets come from environment variables prefixed with SECRET_ and must be base64-encoded.

Quick start

Start Kestra with the four secrets above as SECRET_ environment variables.
Import this flow and run it with the namespace you want to watch.
Create a test deployment with a bad image (kubectl create deployment dw-bad --image=nginx:nonexistent) and run again: the Slack alert lists the deployment and the unavailable replicas.
Run once more without changing anything: no second alert. Delete the deployment and run again: quiet, and the state is cleared.

How to extend

Add per-deployment ignore lists by filtering names in the Python step.
Lower the schedule to every hour for faster detection by changing the cron trigger.
Deploy one copy of the flow per namespace you want to watch, changing only the namespace input.
Route alerts to PagerDuty or Opsgenie by swapping the Slack task for another notification task.

Inputs

Input	Type	Default	What it does
`namespace`	STRING	`default`	the namespace whose deployments to watch
`slack_webhook`	STRING	(required)	Incoming webhook URL of the Slack channel for the alerts

Pitfalls

A deployment that is still rolling out normally shows unavailable replicas for a short time. This flow alerts on the first sighting, so very fast rollouts can cause a one-time alert. The dedup keeps it to one.
The ProgressDeadlineExceeded reason only appears after the progress deadline (default 10 minutes), so the unavailable-replicas check is the faster signal.
Make sure the MASTERURL is reachable from inside the Kestra container. localhost inside the container is not your host: use host.docker.internal when Kestra runs in Docker on the same machine as the cluster.