Page Discord When Prometheus Targets Go Down

The up metric is the heartbeat of every Prometheus deployment, one series per scrape target, 0 when the scrape fails. This blueprint turns that heartbeat into an event. The io.kestra.plugin.prometheus.Trigger evaluates up == 0 every minute and starts an execution only when the query returns at least one result, so nothing runs while the fleet is healthy. When targets drop, the flow pages Discord with {{ trigger.size }}, the number of down targets, giving on-call the blast radius in the first line of the message.

How it works

1. watch_targets (io.kestra.plugin.prometheus.Trigger) polls Prometheus on a PT1M interval with the PromQL query up == 0 and fetchType: FETCH. The trigger fires only when the result set is non-empty, and its outputs mirror the Query task, so the matched series are available as {{ trigger.rows }} and the count as {{ trigger.size }}.
2. page_discord posts the count and the execution id to a Discord channel through an incoming webhook.
3. The errors block posts a distinct Discord message if the alert flow itself fails, so a broken webhook never silently swallows an outage.
4. The trigger ships with disabled: true; flip it once your secrets are in place.

What you get

• An alert that costs zero executions while everything is up.
• The number of down targets in the page, not just a binary alarm.
• A meta-alert covering the failure of the alerting path itself.
• A pattern that generalizes to any PromQL expression, not just up.

Who it's for

• SRE and platform teams who want target-down notifications without running Alertmanager.
• Teams already orchestrating with Kestra who prefer alert logic version-controlled next to their flows.
• Anyone who needs an alert to kick off remediation tasks, not just send a message.

Why orchestrate this with Kestra

Alertmanager sends notifications; Kestra starts workflows. Because the trigger produces a real execution with the query results attached, the response can grow from a Discord message into a remediation pipeline, restart a service, open a ticket, capture diagnostics, all in the same flow, with every firing recorded in the execution history for postmortems.

Prerequisites

• A Prometheus server reachable from your Kestra instance.
• A Discord incoming webhook for the alerts.
• The trigger supports HTTP basic auth through its username and password properties if Prometheus sits behind one.

Secrets

• PROMETHEUS_URL: base URL of the Prometheus server, e.g. http://prometheus:9090.
• DISCORD_WEBHOOK_URL: Discord incoming webhook URL.

Quick start

1. Add the PROMETHEUS_URL and DISCORD_WEBHOOK_URL secrets to your Kestra namespace.
2. Set disabled: false on the watch_targets trigger.
3. Stop one exporter, wait a minute, and check the Discord page reports one target down.

How to extend

• Scope the query to what matters, for example up{job="api"} == 0, to avoid paging on lab instances.
• Raise the interval to PT5M for less urgent environments.
• Add remediation tasks after page_discord, such as restarting the target through an SSH or Kubernetes task.
• Swap the PromQL for any condition worth waking someone for, high error rates, full disks, or stalled queues.

Links

• Prometheus plugin documentation
• Prometheus up metric documentation