Schedule icon
Commands icon
Docker icon
Script icon
If icon
SlackIncomingWebhook icon
Set icon

StuckJobWatch - Kubernetes Jobs past SLA watcher

Alerts on Slack when Kubernetes Jobs run longer than their SLA, with elapsed time and a log tail. Never kills anything.

Categories
Infrastructure

How it works

  1. list_jobs (io.kestra.plugin.scripts.shell.Commands) dumps the namespace's jobs as JSON with kubectl, then grabs the last log lines of every currently active job.
  2. find_stuck (io.kestra.plugin.scripts.python.Script) keeps jobs with status.active whose startTime is older than max_duration_minutes. Each one gets its elapsed time and its log tail attached. It compares against the KV state so only newly stuck jobs alert.
  3. maybe_alert (io.kestra.plugin.core.flow.If) posts the Slack alert only when there is something new: the job, how long it has run past the SLA, and the log tail for the first clue.
  4. save_state (io.kestra.plugin.core.kv.Set) writes back the current stuck list. Finished or deleted jobs drop out on their own, so a job getting stuck again alerts again, and steady-state reruns stay quiet.

What you get

  • A thirty-minute check for runaway Jobs: the migration that never finished, the backup that hung at 3am.
  • Slack alerts with the two things you need first: how long it has been running, and what it was doing last.
  • No alert spam: the KV-backed memory remembers who was already reported.
  • Quiet runs when every job finishes inside its SLA.

Who it's for

Anyone running batch Jobs on Kubernetes: data pipelines, backups, migrations. If a Job has ever silently run for six hours, this is the watcher for it. Works on any cluster you can reach with a client certificate: minikube, kind, managed clusters.

Why orchestrate this with Kestra

A stuck Job is invisible until someone notices the dashboard or the downstream data never arrives. As a flow it checks on a schedule, keeps every run's findings where you can audit them, and only bothers you when something new gets stuck. It completes the Job trilogy with JobWatch (failed Jobs) and JobJanitor (completed-Job cleanup): this one catches Jobs that never finish at all.

Prerequisites

  • A Kubernetes cluster with jobs you can list and read logs for.
  • A kubeconfig client certificate and key with permission to list jobs and pods in the target namespace.
  • Kestra with the Docker task runner available (used for the kubectl and Python steps).
  • A Slack Incoming Webhook URL.

Secrets

Secret What it is
CLIENT_CRT base64-encoded client certificate from your kubeconfig
CLIENT_KEY base64-encoded client key from your kubeconfig
MASTERURL base64-encoded Kubernetes API server URL, reachable from inside the Kestra container
SLACK_WEBHOOK base64-encoded Slack Incoming Webhook URL

In Kestra OSS, secrets come from environment variables prefixed with SECRET_ and must be base64-encoded. If the API URL uses 127.0.0.1 or localhost, replace it with host.docker.internal so the Kestra container can reach it.

Quick start

  1. Start Kestra with the secrets above as SECRET_ environment variables. If minikube restarted since your last run, regenerate SECRET_MASTERURL first: the API port changes on every minikube start.
  2. Import this flow and create a stuck job: kubectl create job stuck-demo --image=busybox -- sh -c 'echo starting work; sleep 3600'. Run the flow with max_duration_minutes: 5: the Slack alert names it with the elapsed time and the log tail.
  3. Run again: quiet, the job is already reported.
  4. Delete the job and run again: quiet, it dropped out of state.

How to extend

  • Watch all namespaces by looping over kubectl get namespaces output in the shell step.
  • Tune the SLA per job with an annotation like stuckjobwatch/max-minutes read in the Python step.
  • Page instead of Slack-alert by swapping the notification task for PagerDuty or Opsgenie.
  • Alert on jobs nearing the SLA (80 percent) as an early warning, not just past it.

Inputs

Input Type Default What it does
namespace STRING default the namespace whose jobs to watch
max_duration_minutes INT 60 jobs active longer than this count as stuck
log_tail_lines INT 20 how many log lines to include per stuck job
slack_webhook STRING (required) Incoming webhook URL of the Slack channel for alerts

Links

Pitfalls

  • This flow deliberately never kills stuck jobs. Killing a Job is a judgment call: the "stuck" migration might be 90 percent done, and deleting it could mean starting over or leaving partial writes. The alert tells you; you decide.
  • max_duration_minutes is per namespace in this flow. Jobs with legitimately different runtimes need the per-job annotation from "How to extend".
  • A Job with no logs yet reports "(no logs yet)". That usually means the pod is still starting, not that anything is broken.
  • The flow is strictly read-only. It lists jobs and reads logs; it never deletes, suspends, or patches anything.
  • Minikube's API server port changes on every minikube start. Regenerate SECRET_MASTERURL and restart the Kestra container, or the kubectl step fails to connect.
See How

New to Kestra?

Use blueprints to kickstart your first workflows.