Query icon
Request icon
OutputValues icon
Log icon
If icon
SlackIncomingWebhook icon
Pause icon
Push icon
Sleep icon
Fail icon
Trigger icon
Schedule icon

Repair stale Pushgateway labels for a live Kestra batch job

Detect stale Pushgateway groups, match live Kestra executions, replace drifted labels and verify the complete metric set.

Categories
Infrastructure

Safety

A Pushgateway group has no expiry. A renamed batch flow can leave a successful timestamp under old labels, so freshness alerts either page forever or read a value that no longer describes the live flow. This flow repairs one stable group at a time.

Guard What it stops
Stale push_time_seconds query Writing a group that is not stale
Recent successful Kestra execution Treating a retired flow as a live repair target
Complete metric and label read Replacing a partial or already-correct group without evidence
Pause plus auto_remediate and dry_run gates An unattended Pushgateway write
Fresh reads after approval Writing from a stale approval snapshot
Query verification of all series and labels Reporting success when Push did not replace the group

The Prometheus plugin has no group-delete operation. Retired groups are therefore reported with no delete attempt. The write path is limited to a live group with drifted metric labels.

How it works

  1. detect_stale_push, read_success_metric, read_last_success_metric and read_duration_metric use io.kestra.plugin.prometheus.Query to read the Pushgateway timestamp and the complete batch metric set.
  2. recent_executions (io.kestra.plugin.core.http.Request) searches Kestra for recent successful executions of the intended flow in the exact namespace.
  3. plan (io.kestra.plugin.core.output.OutputValues) records the stale group, live execution evidence and drifted labels. route continues only when all three are present. A healthy group, a missing execution and a lookalike flow are spared.
  4. approval (io.kestra.plugin.core.flow.Pause) is reached only when auto_remediate is true and dry_run is false. recheck_* tasks read Prometheus and Kestra again after resume. Any changed eligibility fact fails before the write.
  5. replace_group (io.kestra.plugin.prometheus.Push) overwrites those metrics in the stable group with job_last_success_timestamp, job_success and job_duration_seconds under the expected namespace and flow labels.
  6. The verify_* Query tasks require every series, a fresh push_time_seconds and zero old flow-label series. The execution fails when any assertion is false.
  7. stale_push_event (io.kestra.plugin.prometheus.Trigger) is the event path. hourly_backstop is disabled and forces safe inputs.

What you get

  • A live-versus-retired decision grounded in Kestra execution history.
  • A report-only plan that shows stale, missing and drifted series before a write.
  • A rewrite of the three batch metrics through the Prometheus Push plugin.
  • A second read that proves fresh push time and removal of the old labels.
  • Slack alerts on findings and failures with the Kestra execution id.

Who it's for

  • Platform teams that use Pushgateway for short-lived Kestra batch jobs.
  • SRE teams whose freshness alerts are confused by renamed flows.
  • Operators who need a replayable approval record for metric repair.

Why orchestrate this with Kestra

Prometheus can detect an old push but it cannot know whether the producing flow is still live. Kestra joins the gateway series with its own execution history, pauses before the group replacement and retains the full plan, approval and verification in one execution. A shell script would need to reproduce those checks and could not use the Prometheus Push plugin’s write contract.

Tested end to end

Tested on Kestra 2.0.5 with io.kestra.plugin.prometheus against pinned Prometheus and Pushgateway test services. The fixture seeds one group with old flow labels, a recent successful Kestra execution and the three required metrics. The healthy control has a fresh push. The degraded dry run finds the stale group and writes nothing. Declined approval leaves all series unchanged. Remediation replaces all three series and the independent Prometheus API read measures the old-label series dropping from 3 to 0 while the fresh-label series rise from 0 to 3. Repeat is idempotent because the group is fresh. A changed-label race during Pause fails before Push. An ineffective Push response fails the verification query.

Case Execution Result
Healthy 3dMHkoaksa6X67oBId01wV SUCCESS. Fresh group, 3 series spared
Dry run 5ewq2I5I1eRzm81ph95l2e SUCCESS. Stale group found, 3 old-label series before and after
Declined approval 3snewcK5b3STBbH8gkpVT9 SUCCESS. 3 old-label series kept, no new group
Remediation 4aoxV3sbDCVM2zPRE73N88 SUCCESS. Old-label series 3 to 0, correct-label series 0 to 3 with a fresh push_time_seconds
Repeat 1z0iFL2BkiavgChwmsZEj0 SUCCESS. 3 correct series, no stale candidate, no write
Refused push 3iawkAUNW3u6CnlSV947pS FAILED as intended. 3 old-label series remain
Change during the Pause lpxCJz1C2BvBWgPp0AU3w FAILED as intended. 3 old-label series remain, no new group
Second fixture: remediation 3bCqqdaJpNayVRkcW8dINZ SUCCESS. Old 3 to 0, correct 0 to 3

Prerequisites

  • Prometheus must scrape Pushgateway with honor_labels: true and expose push_time_seconds.
  • The Kestra API user needs execution search permission. The Pushgateway credentials need write permission for the stable group.
  • The worker must reach Prometheus, Pushgateway and the Kestra API over the configured URLs.

Secrets

  • PROMETHEUS_URL: Prometheus URL used by the event trigger.
  • KESTRA_PASSWORD: password for the Kestra execution search.
  • SLACK_WEBHOOK_URL: Slack incoming webhook for findings and failures.

Inputs

Input Type Default Purpose
prometheus_url STRING http://prometheus:9090 Prometheus query endpoint
pushgateway_url STRING http://pushgateway:9091 Push endpoint
kestra_url STRING http://localhost:8080 Execution history API
namespace STRING company.team Exact Kestra namespace
expected_flow_id STRING prometheus-batch-job-metrics Live flow label and execution filter
group_job STRING prometheus-batch-job-metrics Stable Pushgateway grouping key
stale_after_seconds INT 3600 Staleness threshold
execution_lookback_hours INT 24 Live execution window
auto_remediate BOOL false Enable the write path
dry_run BOOL true Keep the run read-only

Quick start

  1. Create PROMETHEUS_URL, KESTRA_PASSWORD and SLACK_WEBHOOK_URL secrets in the Kestra namespace.
  2. Set prometheus_url, pushgateway_url, kestra_url, namespace, expected_flow_id and group_job to the fixture or production values.
  3. Execute with auto_remediate=false and dry_run=true. Confirm the plan lists the stale group and the expected live execution.
  4. Set both gates for remediation then resume the Pause with approved=true. Confirm the verification task reports three fresh expected series and zero drifted series.
  5. Enable stale_push_event after a manual dry run proves the selectors. Keep hourly_backstop disabled.

How to extend

  • Add another fixed metric to the Push list and to the pre-write and post-write Query checks.
  • Keep the group job and instance stable when a flow is renamed.
  • Scope the execution filter to a deployment label when several live flows share a namespace.

Limits: the flow does not delete retired groups. It writes the three documented metrics with a POST, so other metric names in the group are kept. It uses the most recent successful execution only as liveness evidence. It does not prove that a successful execution produced the metric values itself.

Links

This blueprint was created by zkasuran.

See How

New to Kestra?

Use blueprints to kickstart your first workflows.