Jenkins Flaky Test Triage and Auto-Retry Gate with Kestra

id: jenkins-flaky-test-triage-and-auto-retry-gate
namespace: company.team

description: |
  On every Jenkins build-finished notification, pull the authoritative build JSON straight from
  the Jenkins REST API, tell a flaky test failure apart from a genuine break and from a
  performance regression on an otherwise-green build, and only ever auto-retry the one case
  where retrying is actually safe: a small minority of tests failing on a build that is
  otherwise passing.

inputs:
  - id: job_name
    type: STRING
    displayName: Jenkins job name
    description: Populated from the webhook on a real fire; set for manual testing.
    defaults: demo-job

  - id: build_number
    type: STRING
    displayName: Build number
    description: Populated from the webhook on a real fire; set for manual testing.
    defaults: "1"

  - id: build_parameters
    type: JSON
    displayName: Original build parameters
    description: Populated from the webhook's build.parameters on a real fire, so an
      auto-retry re-runs the job with the same parameters it originally ran
      with.
    defaults: "{}"

  - id: jenkins_server_url
    type: STRING
    displayName: Jenkins server URL
    defaults: http://localhost:8080

  - id: jenkins_username
    type: STRING
    displayName: Jenkins username
    defaults: admin

  - id: flaky_fail_ratio_threshold
    type: FLOAT
    displayName: Flaky fail ratio threshold
    description: An UNSTABLE build is only classified as flaky when its failing-test
      ratio is at or below this fraction. Above it, too many tests failed to
      call it flaky.
    defaults: 0.1

  - id: slow_regression_multiplier
    type: FLOAT
    displayName: Slow regression multiplier
    description: A SUCCESS build is flagged as a performance regression when its
      duration exceeds Jenkins's own estimatedDuration by this multiple.
    defaults: 1.5

  - id: confirm_auto_retry
    type: BOOL
    displayName: Confirm automatic retry
    description: When true, this flow is authorized to re-trigger the same job, with
      its original parameters, the moment it classifies a build as flaky. When
      false, a flaky build only alerts Slack.
    defaults: false

tasks:
  - id: fetch_build_detail
    type: io.kestra.plugin.core.http.Request
    description: The webhook notification only carries name/number/status, so this
      calls Jenkins's own /api/json endpoint directly for the full build JSON,
      the only place the per-test-suite breakdown
      (actions[].failCount/totalCount) and timing fields live.
    uri: "{{ inputs.jenkins_server_url }}/job/{{ inputs.job_name }}/{{
      inputs.build_number }}/api/json"
    method: GET
    headers:
      Authorization: "Basic {{ (inputs.jenkins_username ~ ':' ~
        secret('JENKINS_API_TOKEN')) | base64encode }}"
    options:
      allowFailed: true

  - id: evaluate
    type: io.kestra.plugin.core.output.OutputValues
    description: Classifies the build from its Jenkins result plus the JUnit test
      action's fail ratio (when the job publishes test results) and its duration
      against Jenkins's own estimate, folding confirm_auto_retry into the flaky
      classification itself.
    values:
      result: "{{ outputs.fetch_build_detail.code == 200 ?
        ((outputs.fetch_build_detail.body | jq('.result // \"UNKNOWN\"')) |
        first) : 'UNREACHABLE' }}"
      duration: "{{ outputs.fetch_build_detail.code == 200 ?
        ((outputs.fetch_build_detail.body | jq('.duration // 0')) | first |
        number) : 0 }}"
      estimated_duration: "{{ outputs.fetch_build_detail.code == 200 ?
        ((outputs.fetch_build_detail.body | jq('.estimatedDuration // 0')) |
        first | number) : 0 }}"
      fail_ratio: >-
        {{ outputs.fetch_build_detail.code != 200 ? 0 :
        ((outputs.fetch_build_detail.body | jq('(([ (.actions // [])[] |
        select(._class == "hudson.tasks.junit.TestResultAction") ] | first) //
        {"failCount":0,"totalCount":0}) as $t | (if $t.totalCount > 0 then
        ($t.failCount / $t.totalCount) else 0 end)')) | first | number) }}
      fail_counts: >-
        {{ outputs.fetch_build_detail.code != 200 ? '0/0' :
        ((outputs.fetch_build_detail.body | jq('(([ (.actions // [])[] |
        select(._class == "hudson.tasks.junit.TestResultAction") ] | first) //
        {"failCount":0,"totalCount":0}) as $t | ($t.failCount | tostring) + "/"
        + ($t.totalCount | tostring)')) | first) }}
      severity: >-
        {{ outputs.fetch_build_detail.code != 200 ? 'BUILD_FAILED' :
        (((outputs.fetch_build_detail.body | jq('.result // "UNKNOWN"')) |
        first) == 'SUCCESS' ? (((((outputs.fetch_build_detail.body |
        jq('.estimatedDuration // 0')) | first | number) > 0) ?
        (((outputs.fetch_build_detail.body | jq('.duration // 0')) | first |
        number) / ((outputs.fetch_build_detail.body | jq('.estimatedDuration //
        0')) | first | number)) : 1) > inputs.slow_regression_multiplier ?
        'SLOW_REGRESSION' : 'HEALTHY') : ((((outputs.fetch_build_detail.body |
        jq('.result // "UNKNOWN"')) | first) == 'UNSTABLE') ?
        (((outputs.fetch_build_detail.body | jq('(([ (.actions // [])[] |
        select(._class == "hudson.tasks.junit.TestResultAction") ] | first) //
        {"failCount":0,"totalCount":0}) as $t | (if $t.totalCount > 0 then
        ($t.failCount / $t.totalCount) else 0 end)')) | first | number) <=
        inputs.flaky_fail_ratio_threshold ? (inputs.confirm_auto_retry ?
        'FLAKY_TESTS_AUTO_RETRY' : 'FLAKY_TESTS_NEEDS_HUMAN') : 'BROKEN_TESTS')
        : 'BUILD_FAILED')) }}

  - id: route
    type: io.kestra.plugin.core.flow.Switch
    description: HEALTHY and SLOW_REGRESSION never retry, a pass is a pass.
      FLAKY_TESTS_AUTO_RETRY is the only branch that re-runs the job, and only
      because the fail ratio already cleared flaky_fail_ratio_threshold and
      confirm_auto_retry authorized it. BROKEN_TESTS and BUILD_FAILED alert
      without retrying, since re-running those would mask a real break.
    value: "{{ outputs.evaluate.values.severity }}"
    cases:
      "HEALTHY":
        - id: log_healthy
          type: io.kestra.plugin.core.log.Log
          message: "{{ inputs.job_name }} #{{ inputs.build_number }} SUCCESS in {{
            outputs.evaluate.values.duration }}ms, no regression."
      "SLOW_REGRESSION":
        - id: alert_slow_regression
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          url: "{{ secret('SLACK_WEBHOOK_URL') }}"
          payload: |
            {{ {'text': 'SLOW_REGRESSION: ' ~ inputs.job_name ~ ' #' ~ inputs.build_number ~ ' passed but took ' ~ outputs.evaluate.values.duration ~ 'ms against a Jenkins-estimated ' ~ outputs.evaluate.values.estimated_duration ~ 'ms. It passed, so nothing was retried; worth a look before it gets worse.'} | toJson }}
      "FLAKY_TESTS_NEEDS_HUMAN":
        - id: alert_flaky_needs_human
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          url: "{{ secret('SLACK_WEBHOOK_URL') }}"
          payload: |
            {{ {'text': 'FLAKY_TESTS_NEEDS_HUMAN: ' ~ inputs.job_name ~ ' #' ~ inputs.build_number ~ ' is UNSTABLE with ' ~ outputs.evaluate.values.fail_counts ~ ' tests failing (ratio ' ~ outputs.evaluate.values.fail_ratio ~ '), at or below the flaky threshold of ' ~ inputs.flaky_fail_ratio_threshold ~ '. Set confirm_auto_retry to true to let this flow re-run it automatically next time, or re-run it manually now.'} | toJson }}
      "FLAKY_TESTS_AUTO_RETRY":
        - id: retry_build
          type: io.kestra.plugin.jenkins.JobBuild
          description: Re-triggers the same job with its original build parameters now
            that confirm_auto_retry authorizes it. Only reachable when the fail
            ratio already cleared flaky_fail_ratio_threshold, so a build with
            too many failures is never auto-retried here.
          jobName: "{{ inputs.job_name }}"
          serverUrl: "{{ inputs.jenkins_server_url }}"
          username: "{{ inputs.jenkins_username }}"
          apiToken: "{{ secret('JENKINS_API_TOKEN') }}"
          parameters: "{{ inputs.build_parameters }}"
        - id: alert_flaky_auto_retried
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          url: "{{ secret('SLACK_WEBHOOK_URL') }}"
          payload: |
            {{ {'text': 'FLAKY_TESTS_AUTO_RETRY: ' ~ inputs.job_name ~ ' #' ~ inputs.build_number ~ ' was UNSTABLE with ' ~ outputs.evaluate.values.fail_counts ~ ' tests failing, under the flaky threshold. Re-triggered the same job automatically with its original parameters.'} | toJson }}
      "BROKEN_TESTS":
        - id: alert_broken_tests
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          url: "{{ secret('SLACK_WEBHOOK_URL') }}"
          payload: |
            {{ {'text': 'BROKEN_TESTS: ' ~ inputs.job_name ~ ' #' ~ inputs.build_number ~ ' is UNSTABLE with ' ~ outputs.evaluate.values.fail_counts ~ ' tests failing, above the flaky threshold of ' ~ inputs.flaky_fail_ratio_threshold ~ '. Too many failures to treat as flaky; this needs a human, not a retry.'} | toJson }}
      "BUILD_FAILED":
        - id: alert_build_failed
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          url: "{{ secret('SLACK_WEBHOOK_URL') }}"
          payload: |
            {{ {'text': 'BUILD_FAILED: ' ~ inputs.job_name ~ ' #' ~ inputs.build_number ~ ' result ' ~ outputs.evaluate.values.result ~ ' (HTTP ' ~ outputs.fetch_build_detail.code ~ ' from Jenkins). Not retried automatically; a hard failure, abort, or unreachable server should be investigated, not masked by a re-run.'} | toJson }}

  - id: summary
    type: io.kestra.plugin.core.log.Log
    message: "Jenkins flaky test triage complete for {{ inputs.job_name }} #{{
      inputs.build_number }}: severity {{ outputs.evaluate.values.severity }},
      fail ratio {{ outputs.evaluate.values.fail_ratio }}."

errors:
  - id: alert_flow_failure
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {{ {'text': 'Jenkins flaky test triage and auto-retry gate failed (execution ' ~ execution.id ~ '). Check the logs.'} | toJson }}

triggers:
  - id: on_build_finished
    type: io.kestra.plugin.core.trigger.Webhook
    description: Jenkins's own free, open-source Notification Plugin posts here when
      a build reaches the FINALIZED phase, carrying the job name, build number,
      status, and the parameters that build ran with.
    key: "{{ secret('JENKINS_NOTIFICATION_WEBHOOK_KEY') }}"
    inputs:
      job_name: "{{ trigger.body.name }}"
      build_number: "{{ trigger.body.build.number }}"
      build_parameters: "{{ trigger.body.build.parameters ?? {} }}"

A Jenkins build that comes back UNSTABLE is not the same problem as one that comes back FAILURE, but most CI dashboards treat them the same way: red is red. A build where 2 of 250 tests failed because of timing flakiness deserves a quiet, automatic re-run. A build where 180 of 250 tests failed does not, that is a real break and a retry would just burn CI capacity while hiding the signal. A build that passed but took three times as long as usual is a third problem entirely, a performance regression nobody will notice until it is much worse. This blueprint reacts to every Jenkins build-finished notification, re-fetches the authoritative build JSON, tells these three cases apart using Jenkins's own result and test action fields, and only ever auto-retries the one case where a retry is actually sound.

This blueprint was created by Mohit Jeswani.

How it works

on_build_finished (io.kestra.plugin.core.trigger.Webhook) receives Jenkins's Notification Plugin payload and maps name, build.number, and build.parameters onto flow inputs, so a manual run with the defaults also works for testing.
fetch_build_detail (io.kestra.plugin.core.http.Request) calls Jenkins's own /job/{name}/{number}/api/json directly with a Basic Auth header, since the webhook payload itself does not carry the JUnit test action's failCount/totalCount or the estimatedDuration field this flow needs, and allowFailed: true keeps a non-200 (for example a build that was deleted before this flow got to it) from crashing the execution.
evaluate (io.kestra.plugin.core.output.OutputValues) pulls the hudson.tasks.junit.TestResultAction entry out of actions with a jq filter (falling back to zero counts when the job publishes no test results), computes the failing-test ratio and the duration-vs-estimate ratio, and classifies the build as HEALTHY, SLOW_REGRESSION, FLAKY_TESTS_AUTO_RETRY, FLAKY_TESTS_NEEDS_HUMAN, BROKEN_TESTS, or BUILD_FAILED, folding confirm_auto_retry into the flaky classification itself.
route (io.kestra.plugin.core.flow.Switch):
- HEALTHY: logs the pass, nothing else happens.
- SLOW_REGRESSION: alerts Slack about the duration drift; no retry, it passed.
- FLAKY_TESTS_AUTO_RETRY: io.kestra.plugin.jenkins.JobBuild re-triggers the same job with its original parameters (the one safe, tool-specific remediation here), then alerts Slack that it did.
- FLAKY_TESTS_NEEDS_HUMAN: alerts Slack instead, since confirm_auto_retry has not authorized this flow to retry automatically yet.
- BROKEN_TESTS: alerts Slack; the fail ratio is too high to call flaky, so this never retries even if confirm_auto_retry is true.
- BUILD_FAILED: alerts Slack; a hard FAILURE/ABORTED result, or an unreachable Jenkins server, never retries.
summary logs the final severity and fail ratio.
errors alerts Slack if the flow itself fails outright, for example a malformed webhook payload.

What you get

A real distinction between a flaky test, a genuine break, and a performance regression, instead of one pass/fail boolean collapsing all three into "red build."
The only auto-retry Jenkins integration in this repo that is gated on the actual failing- test ratio, not just a status code, so it can never quietly mask a real break by retrying it into a false green.
A retry that reuses the original build's parameters via the webhook's own build.parameters, so a parameterized job gets retried the same way it ran the first time.

Who it's for

Platform and CI teams running self-hosted Jenkins who are tired of flaky-test pages and want the retry decision made on the actual fail ratio, not a human re-reading console output at 2am.
Teams evaluating the Jenkins plugin who want a worked example of JobBuild used as a gated remediation action in an event-driven triage gate, distinct from this repo's existing Jenkins blueprints, which poll lastBuild to wait out a build they just triggered rather than reacting to a finished build's own test and timing detail.

Why orchestrate this with Kestra

Jenkins's Notification Plugin can push an event, but it has no concept of a fail-ratio threshold, no memory of what "normal" duration looks like, and no safe way to decide when a retry helps versus when it just hides a problem. Kestra keeps that policy (flaky_fail_ratio_threshold, slow_regression_multiplier, confirm_auto_retry) as flow inputs, turns one webhook event into a six-way classification, and keeps a full execution history of every build this flow has seen, what it decided, and whether it retried.

Prerequisites

A Jenkins instance with the free, open-source Notification Plugin installed and configured to POST to this flow's webhook URL on job completion.
A Jenkins API token (or password) for a user with permission to read build status and trigger builds.
A Slack incoming webhook URL.

A local Jenkins instance for end-to-end testing can be run via the official open-source jenkins/jenkins:lts Docker image, with a test job configured to publish JUnit results so both the flaky and non-flaky paths can be exercised.

Secrets

JENKINS_API_TOKEN: Jenkins API token or password. Used to build the Basic Auth header for fetch_build_detail and passed directly to retry_build. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_JENKINS_API_TOKEN.
JENKINS_NOTIFICATION_WEBHOOK_KEY: the path segment of this flow's webhook URL; keep it unguessable since anyone who knows it can post a fake build event. In Kestra OSS, provide it as SECRET_JENKINS_NOTIFICATION_WEBHOOK_KEY.
SLACK_WEBHOOK_URL: used by every Slack alert case and the errors block. In Kestra OSS, provide it as SECRET_SLACK_WEBHOOK_URL.

Inputs

Input	Default	Purpose
`job_name`	`demo-job`	Populated from the webhook; set directly for manual testing
`build_number`	`1`	Populated from the webhook; set directly for manual testing
`build_parameters`	`{}`	Populated from the webhook's `build.parameters`, reused by `retry_build`
`jenkins_server_url`	`http://localhost:8080`	Base URL used to build the `/api/json` request
`jenkins_username`	`admin`	Paired with `JENKINS_API_TOKEN` for Basic Auth
`flaky_fail_ratio_threshold`	`0.1`	Fail ratio ceiling for an `UNSTABLE` build to count as flaky
`slow_regression_multiplier`	`1.5`	Duration-vs-estimate ceiling before a passing build counts as a regression
`confirm_auto_retry`	`false`	Authorizes `retry_build` to actually fire for a flaky build

Outputs

{{ outputs.fetch_build_detail.body }}: the full Jenkins build JSON (result, building, duration, estimatedDuration, actions, and everything else Jenkins returns from /api/json), as a raw string that jq parses directly.
{{ outputs.evaluate.values.severity }}: HEALTHY, SLOW_REGRESSION, FLAKY_TESTS_AUTO_RETRY, FLAKY_TESTS_NEEDS_HUMAN, BROKEN_TESTS, or BUILD_FAILED.
{{ outputs.evaluate.values.fail_ratio }} / .fail_counts: the failing-test fraction and an N/M string, both 0/"0/0" when the job publishes no JUnit test results at all.

Pitfalls

The failing-test ratio only exists when the job actually publishes JUnit (or compatible) results via a hudson.tasks.junit.TestResultAction. A job with no test publisher reports fail_counts as 0/0 and can only ever land in HEALTHY, SLOW_REGRESSION, or BUILD_FAILED, never the two flaky states, since there is nothing to compute a ratio from.
fetch_build_detail calls /api/json at Jenkins's default tree depth; if a test reporting plugin nests its summary deeper than that default (uncommon, but possible), actions will not contain it and the ratio again falls back to 0/0.
JobBuild's parameters field sends form data; a job with no parameters defined at all should receive build_parameters as {} rather than omitting the field, which is exactly what the webhook mapping's ?? {} fallback produces.
The Notification Plugin's JSON shape (name, build.number, build.phase, build.status, build.parameters) is version-dependent; configure it to notify on phase FINALIZED so this flow only fires once per build, not once per phase transition.

Quick start

Add the JENKINS_API_TOKEN, JENKINS_NOTIFICATION_WEBHOOK_KEY, and SLACK_WEBHOOK_URL secrets.
Install the Notification Plugin on Jenkins and point it at this flow's webhook URL, notifying on phase FINALIZED.
Run a job that publishes JUnit results with everything passing; confirm log_healthy fires.
Make a small minority of tests fail (so Jenkins marks the build UNSTABLE under the ratio threshold) with confirm_auto_retry: false; confirm alert_flaky_needs_human fires.
Set confirm_auto_retry: true and repeat; confirm retry_build re-triggers the job and alert_flaky_auto_retried fires.
Fail the majority of tests instead; confirm alert_broken_tests fires and nothing retries.

How to extend

Track fail ratio history across the last N builds of the same job in a KV pair, and only classify FLAKY_TESTS_AUTO_RETRY if the SAME tests failed inconsistently across builds, not just that this one build's ratio is low.
Add a changeSet check so a build right after a merge is never auto-retried, even if flaky, since a real regression is more likely immediately after a code change.
Cap how many times the same job gets auto-retried per day using a KV counter, so a genuinely broken build under the ratio threshold cannot retry itself indefinitely.

Jenkins Flaky Test Triage and Auto-Retry Gate

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Inputs

Outputs

Pitfalls

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Jenkins Flaky Test Triage and Auto-Retry Gate

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Inputs

Outputs

Pitfalls

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

Odoo Scheduled Action Overdue Sentinel

Ollama Model Storage Headroom and Cache Eviction Sentinel

Gate OIDC Signing-Key Rollovers with a Pinned JWKS

Ollama GPU VRAM Saturation and Idle Model Eviction Sentinel

Daily Cross-Stack Compliance Audit with OPA

Enforce CI/CD guardrails with OPA (Rego) policy gate

New to Kestra?