PodPulse: Kubernetes pod crash watcher blueprint for Kestra

id: podpulse
namespace: company.team
description: Watches pods in a namespace and alerts on Slack when one keeps
  crashing or restarting too often.

labels:
  project: podpulse
  team: platform

inputs:
  - id: namespace
    type: STRING
    displayName: "Namespace"
    description: the namespace whose pods to watch
    defaults: default
  - id: restart_threshold
    type: INT
    displayName: "Restart threshold"
    description: a pod with at least this many container restarts counts as troubled
    defaults: 5
  - id: slack_webhook
    type: STRING
    displayName: "Slack Webhook URL"
    description: Incoming webhook URL of the Slack channel for the alerts

triggers:
  - id: schedule
    type: io.kestra.plugin.core.trigger.Schedule
    description: checks pod health every morning
    cron: "0 8 * * *"
    timezone: UTC
    inputs:
      namespace: default
      restart_threshold: 5
      slack_webhook: "{{ secret('SLACK_WEBHOOK') }}"

tasks:
  - id: list_pods
    type: io.kestra.plugin.scripts.shell.Commands
    description: lists all pods in the namespace as JSON
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
      user: "0"
    containerImage: bitnami/kubectl
    inputFiles:
      client.crt: "{{ secret('CLIENT_CRT') }}"
      client.key: "{{ secret('CLIENT_KEY') }}"
    commands:
      - kubectl get pods -n {{ inputs.namespace }} -o json
        --client-certificate=client.crt --client-key=client.key --server={{
        secret('MASTERURL') }} --insecure-skip-tls-verify=true > pods.json
    outputFiles:
      - pods.json

  - id: find_trouble
    type: io.kestra.plugin.scripts.python.Script
    description: finds crashlooping and flapping pods, drops ones already alerted on
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
      user: "0"
    containerImage: python:3.11
    dependencies:
      - kestra
    script: |
      import json
      from kestra import Kestra

      threshold = int("""{{ inputs.restart_threshold }}""")
      namespace = """{{ inputs.namespace }}"""

      with open("""{{ outputs.list_pods.outputFiles['pods.json'] }}""") as f:
          pods = json.load(f)['items']

      # names we already alerted on, from the KV store (null on first run -> empty)
      raw_state = """{{ kv('podpulse-state-' ~ inputs.namespace, errorOnMissing=false) }}"""
      already = set(json.loads(raw_state) if raw_state.strip() else [])

      troubled = []
      for pod in pods:
          meta = pod.get('metadata', {})
          status = pod.get('status', {})
          name = meta['name']
          if status.get('phase') == 'Succeeded':
              continue  # finished its job, not our problem
          reasons = []
          for cs in status.get('containerStatuses', []):
              cname = cs.get('name')
              waiting = cs.get('state', {}).get('waiting', {})
              if waiting.get('reason') == 'CrashLoopBackOff':
                  reasons.append(f"crashlooping ({cname}: CrashLoopBackOff)")
              restarts = cs.get('restartCount', 0)
              if restarts >= threshold:
                  reasons.append(f"{restarts} restarts ({cname}, threshold {threshold})")
          if reasons:
              troubled.append({'name': name, 'reasons': reasons})

      troubled_names = set(t['name'] for t in troubled)
      new_trouble = [t for t in troubled if t['name'] not in already]
      # the new state is simply who is troubled right now;
      # recovered pods drop out on their own, so a future crash alerts again

      lines = [f"*PodPulse - troubled pods in '{namespace}'*", ""]
      for t in new_trouble:
          lines.append(f"- *{t['name']}*: " + "; ".join(t['reasons']))
          lines.append(f"  inspect: `kubectl describe pod {t['name']} -n {namespace}`")
          lines.append(f"  logs: `kubectl logs {t['name']} -n {namespace}`")
      lines.append("")
      alert_text = "\n".join(lines)

      Kestra.outputs({
          'troubled': new_trouble,
          'count': len(new_trouble),
          'new_state_json': json.dumps(sorted(troubled_names)),
          'alert_text': alert_text,
      })

  - id: maybe_alert
    type: io.kestra.plugin.core.flow.If
    description: only alerts when there is something new to report
    condition: "{{ outputs.find_trouble.vars.count > 0 }}"
    then:
      - id: send_alert
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        description: posts the troubled pods to Slack
        url: "{{ inputs.slack_webhook }}"
        messageText: "{{ outputs.find_trouble.vars.alert_text }}"

  - id: save_state
    type: io.kestra.plugin.core.kv.Set
    description: remembers who is troubled now, so recovered pods alert again next time
    key: "podpulse-state-{{ inputs.namespace }}"
    value: "{{ outputs.find_trouble.vars.new_state_json }}"
    overwrite: true

How it works

list_pods (io.kestra.plugin.scripts.shell.Commands) runs kubectl get pods in the target namespace and saves the full pod list as JSON. It authenticates with a client certificate and key passed as Kestra secrets, the same pattern the Kubernetes gallery blueprints use.
find_trouble (io.kestra.plugin.scripts.python.Script) reads the pod list and the remembered state from the KV store, then checks every pod. Crashlooping: any container stuck waiting with reason CrashLoopBackOff. Flapping: any container whose restart count reached the restart_threshold input. Pods that finished successfully are skipped because a completed Job is not a problem. Anything already alerted on is dropped.
maybe_alert (io.kestra.plugin.core.flow.If) runs only when there is something new to report, and send_alert posts the troubled pods to Slack with the reason plus kubectl describe and kubectl logs commands to inspect each one.
save_state (io.kestra.plugin.core.kv.Set) writes back the new list of troubled pod names as one JSON value. Recovered pods drop out on their own, so a future crash alerts again.

What you get

A daily pod health check for any namespace, no agent to install inside the cluster.
Slack alerts only for new trouble: crash loops and pods restarting too often, each with the exact kubectl commands to dig in.
No alert spam: the KV-backed memory remembers who was already reported, and forgets pods that recovered.
Quiet runs when everything is healthy.

Who it's for

Anyone running workloads on Kubernetes who wants to know about crash-looping pods without watching kubectl get pods all day. Works on any cluster you can reach with a client certificate: minikube, kind, managed clusters.

Why orchestrate this with Kestra

A shell script on a cron server can list pods, but then you need somewhere to keep the "already alerted" state, somewhere to hold the secrets, and a place to see the run history when an alert looks wrong. Kestra gives you all three: the KV store is the memory, secrets stay out of the YAML, and every run keeps its logs and outputs where you can inspect them.

Prerequisites

A Kubernetes cluster reachable from where Kestra runs, with a client certificate and key for authentication.
Kestra with the Docker task runner available (used for the kubectl and Python steps).
A Slack Incoming Webhook URL.

Secrets

Secret	What it is
`CLIENT_CRT`	base64-encoded client certificate for the cluster
`CLIENT_KEY`	base64-encoded client key for the cluster
`MASTERURL`	base64-encoded Kubernetes API server URL, reachable from inside the Kestra container
`SLACK_WEBHOOK`	base64-encoded Slack Incoming Webhook URL

In Kestra OSS, secrets come from environment variables prefixed with SECRET_ and must be base64-encoded.

Quick start

Start Kestra with the four secrets above as SECRET_ environment variables.
Import this flow and run it with the namespace you want to watch.
Make a pod crash (for example a Deployment with an image that exits immediately) and run again: the Slack alert lists the pod, the reason, and the kubectl commands to inspect it.
Run once more without changing anything: no second alert. Fix the pod and run again: quiet, and the state is cleared.

How to extend

Watch image pull problems too by also flagging containers waiting with ImagePullBackOff.
Lower the schedule to every hour for faster detection by changing the cron trigger.
Deploy one copy of the flow per namespace you want to watch, changing only the namespace input.
Route alerts to PagerDuty or Opsgenie by swapping the Slack task for another notification task.

Inputs

Input	Type	Default	What it does
`namespace`	STRING	`default`	the namespace whose pods to watch
`restart_threshold`	INT	`5`	a pod with at least this many container restarts counts as troubled
`slack_webhook`	STRING	(required)	Incoming webhook URL of the Slack channel for the alerts

Pitfalls

Pod names change when a Deployment rolls out, so the remembered state is per pod name. A redeployed pod that crash-loops again counts as new trouble and alerts again, which is usually what you want.
The restart_threshold counts cumulative restarts since the pod started, not restarts per hour. A long-lived pod that restarted 5 times over a month will flag once, then stay quiet thanks to the dedup.
Make sure the MASTERURL is reachable from inside the Kestra container. localhost inside the container is not your host: use host.docker.internal when Kestra runs in Docker on the same machine as the cluster.