Get icon
Get icon
Script icon
Docker icon
Log icon
If icon
SlackIncomingWebhook icon
Set icon
Schedule icon

Monitor prolonged Kubernetes Pod termination with Slack alerts

Monitor prolonged Kubernetes Pod termination with read-only checks, persistent timers, Slack alerts, repeat suppression and clearing notifications.

Categories
Infrastructure

Track prolonged Pod deletion without modifying the cluster. Finalizers are not required; alerts indicate observed delay, not a diagnosed failure.

How it works

  1. A disabled one-minute schedule lists Pods in one namespace through the Kubernetes plugin.
  2. Python compares deletion metadata with Kestra KV state keyed by cluster and namespace. Timers start at first observation, not at deletionTimestamp.
  3. An If task sends Slack alerts after the threshold and clearing messages when alerted UIDs disappear from a successful read. Successful scans suppress repeats.
  4. KV state commits after notification. Concurrency is one; failures alert separately and do not commit state.

What you get

  • Slack messages with namespace, name, UID, phase, finalizers and an inspection command.
  • Execution summary and report.json from the Python task.
  • Read-only monitoring, including Pods with no finalizers. Clearing means absent from the API, not proof that external cleanup succeeded.
  • A crash between sending and saving can repeat a message. Pods deleted between scans are missed; losing KV state resets timers.

Who it's for

Platform engineers and SREs tracking slow workload shutdowns.

Why orchestrate this with Kestra

Scheduled API reads, persistent KV state, conditional notifications and execution logs keep monitoring and delivery observable together.

Prerequisites

Kestra 2.0 with Kubernetes, Python and Slack plugins; a Docker runner with image/package access; a reachable Kubernetes API; a Slack incoming webhook. Bind a ServiceAccount to a namespaced Role with apiGroups: [""], resources: ["pods"], verbs: ["get", "list"]. Rotate expiring tokens. API hostname must match its TLS certificate.

Secrets

  • POD_WATCH_K8S_URL: HTTPS API endpoint reachable from the Kestra worker.
  • POD_WATCH_K8S_TOKEN: ServiceAccount bearer token.
  • POD_WATCH_K8S_CA_CERT_DATA: Base64-encoded cluster CA certificate.
  • POD_WATCH_SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

  1. Configure the ServiceAccount, RoleBinding and four secrets. Import the blueprint; omit catalog-only extend when pasting YAML directly into the flow editor.
  2. Set stable, unique cluster_id and scan_namespace defaults. Keep one flow responsible for each state key. Choose threshold_seconds for your workloads; 60 seconds is for a short demo.
  3. Execute manually and verify logs. Set the schedule's disabled to false and save. Alert timing depends on scan cadence and execution delays.
  4. Observe a deleting Pod beyond the threshold, then its disappearance. No special application hook is required in production.

How to extend

Add another notification channel or collect Kubernetes events for investigation.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.