GCP Pub/Sub Dead-Letter Backlog Metric Sentinel

A message that keeps failing downstream processing lands in its Pub/Sub dead-letter subscription and then just sits there - Pub/Sub itself has no built-in alert for "something is accumulating in the dead-letter queue." Draining the subscription to count what is in it is destructive: Pub/Sub acknowledgement is permanent, so a monitoring probe that pulls messages just to count them also deletes the evidence a human would need to triage. This blueprint instead reads Cloud Monitoring's own num_undelivered_messages metric for the subscription - the same number Pub/Sub itself already tracks - and alerts on it without ever touching a single message.

This blueprint was created by Mohit Jeswani.

How it works

1. periodic_dlq_backlog_check (io.kestra.plugin.core.trigger.Schedule) runs every 15 minutes, explicitly passing gcp_project_id, dead_letter_subscription, backlog_alert_threshold, and query_window through its own inputs: block so the flow is self-contained when the trigger fires. Shipped disabled so a manual run can be verified first.
2. probe_dlq_backlog_metric (io.kestra.plugin.gcp.monitoring.Query) queries Cloud Monitoring's own pubsub.googleapis.com/subscription/num_undelivered_messages metric, filtered to the dead-letter subscription, over the last query_window. This is a pure metrics read - no Pub/Sub API call is made at all.
3. evaluate_metric_presence (io.kestra.plugin.core.flow.If) branches on whether any time series came back (count).
   • then (NO_DATA): log_no_metric_data flags a likely misconfiguration rather than guessing it means a healthy backlog.
   • else: nested evaluate_backlog_threshold (io.kestra.plugin.core.flow.If) branches on whether the most recent sampled value exceeds backlog_alert_threshold.
     • then (BACKLOG): alert_backlog_detected reports the current count to Slack.
     • else (HEALTHY): log_backlog_healthy records a clean check.
4. log_audit_complete always runs last, printing the time-series count regardless of which branch fired.
5. The errors block alerts Slack separately if the flow itself fails outright - including the unguarded edge case of a time series with zero points, which would otherwise throw while rendering series[0].points[0].

What you get

• A NO_DATA/HEALTHY/BACKLOG answer sourced from Cloud Monitoring's own backlog metric, with zero risk to the messages themselves - nothing is ever pulled or acknowledged.
• A distinct NO_DATA state instead of silently treating a missing metric as "zero backlog," which would hide a misconfigured subscription name or project.
• A worked example of io.kestra.plugin.gcp.monitoring.Query applied to a specific subscription via its resource.labels.subscription_id filter label, not just a generic CPU/latency dashboard query.
• A sentinel that is safe to run as often as you like, since repeated checks cannot themselves cause any data loss - unlike a drain-and-count approach built on io.kestra.plugin.gcp.pubsub.Consume.

Who it's for

• Data platform teams running Pub/Sub dead-letter policies who need an early signal when a consumer starts failing messages into quarantine, without risking the quarantined messages themselves.
• On-call engineers who want one Slack alert with the real backlog count instead of opening the Cloud Monitoring console, or worse, draining the subscription by hand to see what is in it.
• Anyone evaluating the GCP Monitoring plugin who wants a worked example of Query scoped to one specific Pub/Sub subscription via a Cloud Monitoring resource label filter.

Why orchestrate this with Kestra

Cloud Monitoring can display this metric on a dashboard, but it has no first-class "check this subscription on a schedule and tell me HEALTHY or BACKLOG" primitive of its own, and GCP's native alerting policies are a separate configuration surface from the flow that actually needs to react. Kestra supplies the schedule, the NO_DATA/HEALTHY/BACKLOG classification as a first-class branch, and an execution history that shows exactly when a dead-letter subscription started accumulating messages.

Prerequisites

• A GCP project with a Pub/Sub topic whose subscription has a dead-letter policy configured, pointing at dead_letter_subscription (update gcp_project_id, dead_letter_subscription).
• A GCP service account JSON key with roles/monitoring.viewer (read-only; no Pub/Sub permissions are needed at all, since this blueprint never calls the Pub/Sub API).
• A Slack incoming webhook for alerts.

This requires a real GCP project - Cloud Monitoring has no official local emulator of its own, unlike some other GCP services, so there is no fully-offline Docker-only way to exercise this specific blueprint. This is the same baseline prerequisite every other GCP-targeting blueprint in this repository assumes.

Secrets

• GCP_SERVICE_ACCOUNT_JSON: service account key used by io.kestra.plugin.gcp.monitoring.Query. Only needs Monitoring Viewer, deliberately narrower than the Pub/Sub Subscriber role a drain-based approach would require.
• SLACK_WEBHOOK_URL: Slack incoming webhook used by alert_backlog_detected and the errors block.
• In Kestra OSS (no Enterprise secrets backend), secrets are supplied as environment variables prefixed SECRET_, base64-encoded - for example SECRET_SLACK_WEBHOOK_URL=$(echo -n '<your-webhook-url>' | base64) - and read back in flows with {{ secret('NAME') }}.

Inputs

Input	Type	Default	Purpose
gcp_project_id	String	<your-gcp-project-id>	GCP project containing the subscription
dead_letter_subscription	String	orders-dlq-sub	Subscription audited via its subscription_id resource label
backlog_alert_threshold	Integer	0	Undelivered-message ceiling before BACKLOG fires
query_window	String	PT5M	Cloud Monitoring lookback window duration

Outputs

• outputs.probe_dlq_backlog_metric.count: number of matching time series (expected 0 or 1 for a single-subscription filter).
• outputs.probe_dlq_backlog_metric.series: the raw Cloud Monitoring time series; this blueprint reads only series[0].points[0].value.int64Value, the most recent sampled undelivered-message count.

Quick start

1. Create a topic and a dead-letter-policy subscription in a real GCP project, named to match dead_letter_subscription.
2. Add the GCP_SERVICE_ACCOUNT_JSON (Monitoring Viewer role) and SLACK_WEBHOOK_URL secrets.
3. Run the flow manually against a quiet dead-letter subscription and confirm log_backlog_healthy (or log_no_metric_data if the metric has not reported yet) fires.
4. Deliberately fail a message enough times on the primary subscription to trigger its dead-letter policy, wait a minute or two for Cloud Monitoring to sample the metric, then re-run to confirm alert_backlog_detected fires with the real count.
5. Enable periodic_dlq_backlog_check once you trust the check against your real subscription.

How to extend

• Loop over several dead_letter_subscription values with io.kestra.plugin.core.flow.Loop to audit every dead-letter subscription in a project in one run.
• Add a second, separate approval-gated flow using io.kestra.plugin.gcp.pubsub.Consume for the actual replay/triage step once a human has decided to act on a BACKLOG alert - deliberately kept out of this sentinel so the monitoring path itself never touches message data.
• Track the metric value over successive runs (via a Kestra KV store) to alert on a sustained climbing trend even before any single run's count crosses backlog_alert_threshold.

Pitfalls

• This blueprint deliberately does not use io.kestra.plugin.gcp.pubsub.Consume. That task permanently acknowledges every message it pulls (verified from its own source - there is no peek-without-ack option), which would destroy the exact quarantined payloads a human needs to triage. Reading the Cloud Monitoring metric instead keeps this sentinel entirely non-destructive.
• series[0].points[0] is not guarded against a time series with zero points. Cloud Monitoring's own documentation describes points as always present for an active GAUGE metric like this one within a reasonable window, but an edge case here surfaces as a flow failure through the errors block rather than a silent misclassification.
• Cloud Monitoring's protobuf-JSON mapping serializes INT64 values as JSON strings, not numbers - int64Value is cast with the number Pebble filter before the threshold comparison for exactly that reason.
• Cloud Monitoring has no local emulator. Unlike Pub/Sub itself (which does have an official emulator, with its own worker-level environment-variable caveats), there is no way to exercise this specific blueprint without a real GCP project.
• This blueprint is UNTESTED against a live GCP project. The num_undelivered_messages metric type, the subscription_id resource label, and the points[].value.int64Value JSON shape are taken verbatim from Google's own Cloud Monitoring documentation, and the io.kestra.plugin.gcp.monitoring.Query properties and outputs are taken from the plugin's own source on main - but no Kestra engine was run to execute this flow end to end.

Links

• Kestra GCP plugin documentation
• GCP Pub/Sub monitoring metrics (num_undelivered_messages)
• GCP Cloud Monitoring TimeSeries API reference
• Kestra secrets (OSS vs Enterprise)