Schedule icon
Request icon
If icon
SlackIncomingWebhook icon
Log icon

Jenkins Stuck Build Watchdog

Poll Jenkins' own build API, detect hung or failed builds, and alert Slack before a stuck executor wastes an agent slot.

Categories
CoreInfrastructure

A Jenkins build that hangs - a test suite deadlocked, a script waiting on stdin, a zombie process holding a lock - ties up an executor slot indefinitely with no error and no obvious signal in the Jenkins UI unless someone happens to look. This blueprint polls Jenkins' own build API for a job's latest build, classifies it into a four-state result, and alerts Slack on anything short of a clean, finished build.

How it works

  1. periodic_build_check (io.kestra.plugin.core.trigger.Schedule) runs every 15 minutes, explicitly passing jenkins_url, job_name, and max_stuck_minutes through its own inputs: block so the flow is self-contained when the trigger fires. Shipped disabled so a manual run can be verified first.
  2. probe_latest_build (io.kestra.plugin.core.http.Request) calls Jenkins' own lastBuild/api/json endpoint directly, fetching the newest build's complete JSON metadata including building, result, timestamp, and number.
  3. evaluate_still_building (io.kestra.plugin.core.flow.If) branches on building.
    • then (still running): nested evaluate_stuck (io.kestra.plugin.core.flow.If) branches on whether the build's start time is older than max_stuck_minutes ago.
      • then (STUCK): alert_stuck_build reports the build number and start time to Slack.
      • else (RUNNING_OK): log_running_ok records that it is still within the acceptable window.
    • else (finished): nested evaluate_result (io.kestra.plugin.core.flow.If) branches on result == 'SUCCESS'.
      • then (HEALTHY): log_build_healthy records a clean finish.
      • else (FAILED): alert_build_failed reports the actual Jenkins result value (FAILURE, UNSTABLE, or ABORTED) to Slack.
  4. log_audit_complete always runs last, printing the build number, building, and result regardless of which branch fired.
  5. The errors block alerts Slack separately if the flow itself fails outright - an unreachable controller or a missing job must not read as any of the four handled states.

What you get

  • A four-state answer (RUNNING_OK / STUCK / HEALTHY / FAILED) sourced directly from Jenkins' own build API, not inferred from a webhook that a hung build never sends.
  • A distinct STUCK state for a build that is neither finished nor failed - the state a dashboard built only on Jenkins' own pass/fail webhooks cannot see at all.
  • One Slack alert with the exact build number, start time, and (for finished builds) the real Jenkins result string, instead of opening the Jenkins UI to check.
  • Direct querying of Jenkins' native lastBuild/api/json endpoint via io.kestra.plugin.core.http.Request, inspecting runtime properties cleanly with Pebble JSON parsing.

Who it's for

  • Platform and build engineering teams running self-hosted Jenkins who need an early signal when an executor hangs instead of discovering it when a queue backs up.
  • On-call engineers who want one Slack alert with the build number and duration instead of opening the Jenkins UI to check why a job has not finished.
  • Anyone looking to monitor Jenkins build health and executor states automatically without installing third-party Jenkins plugins or managing incoming webhooks.

Why orchestrate this with Kestra

Jenkins' own build page shows one build at a time and sends a webhook only when a build finishes - it has no concept of "this has been running too long" and nothing fires at all for a build that never finishes. Kestra supplies the schedule, the RUNNING_OK/STUCK/HEALTHY/FAILED classification as a first-class branch, and an execution history that shows exactly when a build crossed from merely slow into stuck.

Prerequisites

  • A reachable Jenkins controller with at least one job that has run at least once (update jenkins_url and job_name).
  • A Jenkins user and API token with read access to that job, if the controller has security enabled.
  • A Slack incoming webhook for alerts.

Local testing:

docker run -d --name jenkins-watchdog -p 8080:8081 -p 50000:50000 jenkins/jenkins:lts
docker network connect YOUR_KESTRA_NETWORK jenkins-watchdog

Run docker network ls first to find the active Kestra network name. The container's internal port 8080 is mapped to host port 8081 here specifically so it never collides with Kestra's own UI on host port 8080; open http://localhost:8081 once to run the first-run setup wizard (unlock key from docker logs jenkins-watchdog, create the initial admin user), then create one freestyle job named test with a build step that sleeps long enough to exercise the STUCK path (for example sleep 90) before pointing jenkins_url at the container.

Secrets

  • JENKINS_USERNAME / JENKINS_API_TOKEN: credentials used by probe_latest_build's Basic Auth header. Jenkins' own documentation describes generating an API token under the user's security settings; a plain password also works for Basic Auth but a token is Jenkins' documented preference.
  • SLACK_WEBHOOK_URL: Slack incoming webhook used by alert_stuck_build, alert_build_failed, and the errors block.
  • In Kestra OSS (no Enterprise secrets backend), secrets are supplied as environment variables prefixed SECRET_, base64-encoded - for example SECRET_JENKINS_API_TOKEN=$(echo -n '<your-token>' | base64) - and read back in flows with {{ secret('NAME') }}.

Inputs

  • jenkins_url (STRING, default http://localhost:8080): base URL of the Jenkins controller.
  • job_name (STRING, default test): job name or foldered path to audit.
  • max_stuck_minutes (INT, default 60): running-duration ceiling before a still-building build counts as STUCK.

Outputs

  • outputs.probe_latest_build.body: the full raw Jenkins build JSON string - this blueprint reads building, result, timestamp, and number directly from the response.

Quick start

  1. Start Jenkins locally with the command above and create the test job as described in Prerequisites.
  2. Add the JENKINS_USERNAME, JENKINS_API_TOKEN, and SLACK_WEBHOOK_URL secrets (on a security-disabled local QuickStart, the credentials can be any placeholder value - the header is sent but ignored).
  3. Trigger the test job manually in the Jenkins UI, then immediately run this flow and confirm log_running_ok or alert_stuck_build fires depending on max_stuck_minutes and how long the job's build step sleeps.
  4. Let the job finish successfully and re-run to confirm log_build_healthy fires; make the job fail and re-run to confirm alert_build_failed fires with the real result value.
  5. Enable periodic_build_check once you trust the check against your real job.

How to extend

  • Loop over several job_name values with io.kestra.plugin.core.flow.Loop to watchdog every critical job on a controller in one run.
  • Add an io.kestra.plugin.jenkins.JobBuild task on the STUCK branch to trigger a replacement build on a different node once the stuck one is manually aborted, rather than just alerting.
  • Track duration for the last several HEALTHY builds (via a Kestra KV store) to alert on a build that finished successfully but took dramatically longer than its own history, not just an outright STUCK one.

Pitfalls

  • io.kestra.plugin.jenkins.JobInfo requires a static integer buildNumber and rejects dynamic expressions at parse time in Kestra 2.0.1. This blueprint queries Jenkins' native lastBuild/api/json endpoint directly via io.kestra.plugin.core.http.Request, inspecting the latest build dynamically without static parameter limitations.
  • No verified Kestra Pebble filter returns the current time as epoch milliseconds, so the STUCK comparison converts Jenkins' timestamp (epoch millis) to a UTC ISO-8601 string with jq's todate and compares it against a UTC-formatted now() | dateAdd(...) string instead of doing raw millisecond arithmetic.
  • A freshly created job with no build history yet will fail at probe_latest_build (no lastBuild exists) - this surfaces through the errors block, not as a false RUNNING_OK or HEALTHY.
  • This blueprint is UNTESTED against a live Jenkins controller. The lastBuild/api/json endpoint and the building/result/timestamp build JSON fields are taken from Jenkins' own REST API documentation, but no Docker container or Kestra engine was run to execute this flow end to end.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.