Multi-Tenant CI/CD, Promote Kestra Flows Across Environments and Tenants Automatically
Promote Kestra flows across tenants and environments automatically using the Kestra REST API on a schedule.
Alerts on Slack when Kubernetes Jobs run longer than their SLA, with elapsed time and a log tail. Never kills anything.
list_jobs (io.kestra.plugin.scripts.shell.Commands) dumps the namespace's jobs as JSON with kubectl, then grabs the last log lines of every currently active job.find_stuck (io.kestra.plugin.scripts.python.Script) keeps jobs with status.active whose startTime is older than max_duration_minutes. Each one gets its elapsed time and its log tail attached. It compares against the KV state so only newly stuck jobs alert.maybe_alert (io.kestra.plugin.core.flow.If) posts the Slack alert only when there is something new: the job, how long it has run past the SLA, and the log tail for the first clue.save_state (io.kestra.plugin.core.kv.Set) writes back the current stuck list. Finished or deleted jobs drop out on their own, so a job getting stuck again alerts again, and steady-state reruns stay quiet.Anyone running batch Jobs on Kubernetes: data pipelines, backups, migrations. If a Job has ever silently run for six hours, this is the watcher for it. Works on any cluster you can reach with a client certificate: minikube, kind, managed clusters.
A stuck Job is invisible until someone notices the dashboard or the downstream data never arrives. As a flow it checks on a schedule, keeps every run's findings where you can audit them, and only bothers you when something new gets stuck. It completes the Job trilogy with JobWatch (failed Jobs) and JobJanitor (completed-Job cleanup): this one catches Jobs that never finish at all.
| Secret | What it is |
|---|---|
CLIENT_CRT |
base64-encoded client certificate from your kubeconfig |
CLIENT_KEY |
base64-encoded client key from your kubeconfig |
MASTERURL |
base64-encoded Kubernetes API server URL, reachable from inside the Kestra container |
SLACK_WEBHOOK |
base64-encoded Slack Incoming Webhook URL |
In Kestra OSS, secrets come from environment variables prefixed with SECRET_ and must be base64-encoded. If the API URL uses 127.0.0.1 or localhost, replace it with host.docker.internal so the Kestra container can reach it.
SECRET_ environment variables. If minikube restarted since your last run, regenerate SECRET_MASTERURL first: the API port changes on every minikube start.kubectl create job stuck-demo --image=busybox -- sh -c 'echo starting work; sleep 3600'. Run the flow with max_duration_minutes: 5: the Slack alert names it with the elapsed time and the log tail.kubectl get namespaces output in the shell step.stuckjobwatch/max-minutes read in the Python step.| Input | Type | Default | What it does |
|---|---|---|---|
namespace |
STRING | default |
the namespace whose jobs to watch |
max_duration_minutes |
INT | 60 |
jobs active longer than this count as stuck |
log_tail_lines |
INT | 20 |
how many log lines to include per stuck job |
slack_webhook |
STRING | (required) | Incoming webhook URL of the Slack channel for alerts |
max_duration_minutes is per namespace in this flow. Jobs with legitimately different runtimes need the per-job annotation from "How to extend".minikube start. Regenerate SECRET_MASTERURL and restart the Kestra container, or the kubectl step fails to connect.