Schedule icon
Commands icon
Docker icon
Script icon
If icon
SlackIncomingWebhook icon
Set icon

PVCWatch - Kubernetes stuck PersistentVolumeClaim watcher

A Kestra blueprint that watches Kubernetes PersistentVolumeClaims, detects ones stuck in Pending, and alerts on Slack without spamming.

Categories
Infrastructure

How it works

  1. list_pvcs (io.kestra.plugin.scripts.shell.Commands) runs kubectl get pvc in the target namespace and saves the full claim list as JSON. It authenticates with a client certificate and key passed as Kestra secrets, the same pattern the Kubernetes gallery blueprints use.
  2. find_trouble (io.kestra.plugin.scripts.python.Script) reads the remembered state from the KV store, then checks every claim's phase. Anything stuck in Pending counts as troubled. Anything already alerted on is dropped.
  3. maybe_alert (io.kestra.plugin.core.flow.If) runs only when there is something new to report, and send_alert posts the stuck claims to Slack with the storage class and a kubectl command to inspect each one.
  4. save_state (io.kestra.plugin.core.kv.Set) writes back the new list of troubled claim names as one JSON value. Bound claims drop out on their own, so a future stuck claim alerts again.

What you get

  • A daily storage health check for any namespace, no agent to install inside the cluster.
  • Slack alerts only for new trouble: which claim, which storage class it asked for, and the kubectl command to look at it.
  • No alert spam: the KV-backed memory remembers who was already reported, and forgets claims that got bound.
  • Quiet runs when every claim is bound.

Who it's for

Anyone running stateful workloads on Kubernetes who has watched a pod sit in Pending for hours because its volume never bound. Works on any cluster you can reach with a client certificate: minikube, kind, managed clusters.

Why orchestrate this with Kestra

A shell script on a cron server can list PVCs, but then you need somewhere to keep the "already alerted" state, somewhere to hold the secrets, and a place to see the run history when an alert looks wrong. Kestra gives you all three: the KV store is the memory, secrets stay out of the YAML, and every run keeps its logs and outputs where you can inspect them.

Prerequisites

  • A Kubernetes cluster reachable from where Kestra runs, with a client certificate and key for authentication.
  • Kestra with the Docker task runner available (used for the kubectl and Python steps).
  • A Slack Incoming Webhook URL.

Secrets

Secret What it is
CLIENT_CRT base64-encoded client certificate for the cluster
CLIENT_KEY base64-encoded client key for the cluster
MASTERURL base64-encoded Kubernetes API server URL, reachable from inside the Kestra container
SLACK_WEBHOOK base64-encoded Slack Incoming Webhook URL

In Kestra OSS, secrets come from environment variables prefixed with SECRET_ and must be base64-encoded.

Quick start

  1. Start Kestra with the four secrets above as SECRET_ environment variables.
  2. Import this flow and run it with the namespace you want to watch.
  3. Create a test claim with a bogus storage class and run again: the Slack alert lists the claim and the class it asked for.
  4. Run once more without changing anything: no second alert. Delete the claim and run again: quiet, and the state is cleared.

How to extend

  • Add a stuck-duration threshold by comparing the claim's creation timestamp against now.
  • Lower the schedule to every hour for faster detection by changing the cron trigger.
  • Deploy one copy of the flow per namespace you want to watch, changing only the namespace input.
  • Route alerts to PagerDuty or Opsgenie by swapping the Slack task for another notification task.

Inputs

Input Type Default What it does
namespace STRING default the namespace whose PVCs to watch
slack_webhook STRING (required) Incoming webhook URL of the Slack channel for the alerts

Links

Pitfalls

  • A freshly created claim shows Pending for a short while before it binds. This flow alerts on the first sighting, so very fast provisioning can cause a one-time alert. The dedup keeps it to one.
  • Claims with no storage class stay Pending until a PV is manually created or a default class exists. The alert names the requested class so you can spot the typo.
  • Make sure the MASTERURL is reachable from inside the Kestra container. localhost inside the container is not your host: use host.docker.internal when Kestra runs in Docker on the same machine as the cluster.
See How

New to Kestra?

Use blueprints to kickstart your first workflows.