Recover a Pulumi Stack Wedged by an Interrupted Update
Detect pending operations and a held lock on a Pulumi stack after an interrupted update, then clear them with pulumi cancel and pulumi refresh behind an approval, then verify the stack is clean.
Alerts on Slack when Kubernetes Jobs run longer than their SLA, with elapsed time and a log tail. Never kills anything.
list_jobs (io.kestra.plugin.scripts.shell.Commands) dumps the namespace's jobs as JSON with kubectl, then grabs the last log lines of every currently active job.find_stuck (io.kestra.plugin.scripts.python.Script) keeps jobs with status.active whose startTime is older than max_duration_minutes. Each one gets its elapsed time and its log tail attached. It compares against the KV state so only newly stuck jobs alert.maybe_alert (io.kestra.plugin.core.flow.If) posts the Slack alert only when there is something new: the job, how long it has run past the SLA, and the log tail for the first clue.save_state (io.kestra.plugin.core.kv.Set) writes back the current stuck list. Finished or deleted jobs drop out on their own, so a job getting stuck again alerts again, and steady-state reruns stay quiet.Anyone running batch Jobs on Kubernetes: data pipelines, backups, migrations. If a Job has ever silently run for six hours, this is the watcher for it. Works on any cluster you can reach with a client certificate: minikube, kind, managed clusters.
A stuck Job is invisible until someone notices the dashboard or the downstream data never arrives. As a flow it checks on a schedule, keeps every run's findings where you can audit them, and only bothers you when something new gets stuck. It completes the Job trilogy with JobWatch (failed Jobs) and JobJanitor (completed-Job cleanup): this one catches Jobs that never finish at all.
| Secret | What it is |
|---|---|
CLIENT_CRT |
base64-encoded client certificate from your kubeconfig |
CLIENT_KEY |
base64-encoded client key from your kubeconfig |
MASTERURL |
base64-encoded Kubernetes API server URL, reachable from inside the Kestra container |
SLACK_WEBHOOK |
base64-encoded Slack Incoming Webhook URL |
In Kestra OSS, secrets come from environment variables prefixed with SECRET_ and must be base64-encoded. If the API URL uses 127.0.0.1 or localhost, replace it with host.docker.internal so the Kestra container can reach it.
SECRET_ environment variables. If minikube restarted since your last run, regenerate SECRET_MASTERURL first: the API port changes on every minikube start.kubectl create job stuck-demo --image=busybox -- sh -c 'echo starting work; sleep 3600'. Run the flow with max_duration_minutes: 5: the Slack alert names it with the elapsed time and the log tail.kubectl get namespaces output in the shell step.stuckjobwatch/max-minutes read in the Python step.| Input | Type | Default | What it does |
|---|---|---|---|
namespace |
STRING | default |
the namespace whose jobs to watch |
max_duration_minutes |
INT | 60 |
jobs active longer than this count as stuck |
log_tail_lines |
INT | 20 |
how many log lines to include per stuck job |
slack_webhook |
STRING | (required) | Incoming webhook URL of the Slack channel for alerts |
max_duration_minutes is per namespace in this flow. Jobs with legitimately different runtimes need the per-job annotation from "How to extend".minikube start. Regenerate SECRET_MASTERURL and restart the Kestra container, or the kubectl step fails to connect.