Push Batch Job Metrics to the Prometheus Pushgateway

Prometheus is built around scraping long-running services, which leaves batch jobs invisible unless they push their own telemetry. This blueprint makes batch jobs first-class citizens in Prometheus monitoring. The flow runs a piece of work, then io.kestra.plugin.prometheus.Push records job_duration_seconds and job_success in the Pushgateway, labeled with the Kestra namespace and flow id. When the work fails, the errors block pushes job_success with value 0 and alerts Slack, so a single PromQL alert rule such as job_success == 0 covers every batch job that adopts the pattern.

How it works

1. run_batch_job (io.kestra.plugin.core.debug.Return) stands in for the real workload. Swap it for a Python script, a warehouse query, or a subflow call without touching the rest of the flow.
2. push_success_metrics (io.kestra.plugin.prometheus.Push) sends two metrics to the Pushgateway under the job name {{ flow.id }} and instance kestra. The duration is computed in Pebble as the difference between now() and execution.startDate in epoch seconds, and job_success is pushed as 1.
3. If any task fails, the errors block runs push_failure_metric, pushing job_success 0 with the same labels, then alert_failure posts the flow and execution id to Slack.
4. A disabled-by-default Schedule trigger runs the job nightly at 02:00.

What you get

• Batch jobs visible in Grafana next to your services, with duration trends per flow.
• One reusable PromQL alert, job_success == 0, instead of a bespoke alert per job.
• A failure path that reports twice, once to Prometheus for machines and once to Slack for humans.
• Labels driven by {{ flow.namespace }} and {{ flow.id }}, so copies of this flow never collide in the Pushgateway.

Who it's for

• Platform teams who monitor services with Prometheus but have no visibility into scheduled jobs.
• Data engineers who want pipeline duration and success rates on existing Grafana dashboards.
• SRE teams consolidating batch alerting into the PromQL rules they already operate.

Why orchestrate this with Kestra

Instrumenting a batch script by hand means writing Pushgateway HTTP calls, remembering to push a failure marker in every error path, and repeating that in every job. Kestra separates the concerns declaratively. The workload task stays clean, the Push task handles the metric protocol, and the errors block guarantees the failure marker and the Slack alert fire no matter which task broke.

Prerequisites

• A running Prometheus Pushgateway reachable from your Kestra workers, with Prometheus scraping it.
• A Slack incoming webhook for failure alerts.
• The Push task supports HTTP basic auth through its username and password properties if your Pushgateway sits behind one.

Secrets

• PUSHGATEWAY_URL: base URL of the Pushgateway, e.g. http://pushgateway:9091.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the PUSHGATEWAY_URL and SLACK_WEBHOOK_URL secrets to your Kestra namespace.
2. Execute the flow, then query job_success{flow="prometheus-batch-job-metrics"} in Prometheus to confirm the push arrived.
3. Replace run_batch_job with your real workload and set disabled: false on the nightly trigger.

How to extend

• Add business metrics to the push, such as rows processed or bytes moved, by appending entries to metrics from task outputs.
• Create a Prometheus alert rule on time() - push_time_seconds to catch jobs that stopped running entirely.
• Wrap the pattern around existing flows by calling them with io.kestra.plugin.core.flow.Subflow between the push tasks.
• Pair with the Prometheus SLO gate blueprint to act on the metrics this flow produces.

Links

• Prometheus plugin documentation
• Pushgateway documentation