Maintain message broker health by auditing AMQP queue depth before consumer lag causes memory pressure or downstream delivery timeouts. This blueprint reads the number of unconsumed messages in a target queue from the RabbitMQ management API, without consuming any of them, evaluates backlog metrics against configurable threshold rules, and dispatches automated Slack notifications and safe quarantine logging when queue depth breaches operational SLAs.
How it works
- probe_queue (io.kestra.plugin.core.http.Request) reads the designated queue (inputs.queue_name) from the RabbitMQ management API (GET /api/queues/{vhost}/{queue}). The response holds messages_ready, the number of messages waiting for a consumer. Nothing is consumed or acknowledged, so the queue is left as it was.
- evaluate_backlog (io.kestra.plugin.core.flow.Switch) checks whether messages_ready exceeds the backlog_threshold value.
- If the count exceeds the threshold, dispatch_backlog_alert (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) posts a notification to Slack detailing the queue name, message count, and remediation status.
- evaluate_remediation (io.kestra.plugin.core.flow.If) checks whether remediation_action is set to PURGE_EXPIRED and dry_run is set to false, logging active quarantine remediation if enabled.
- If the count is within threshold, log_queue_healthy (io.kestra.plugin.core.log.Log) records a healthy audit status.
- log_audit_summary (io.kestra.plugin.core.log.Log) confirms completion of the audit pass with full execution lineage.
- A flow-level errors block (alert_flow_failure) catches broker connection failures or unhandled task errors, dispatching an incident notification to Slack.
What you get
- Automated, non-destructive queue depth checks: the flow reads the broker's counters and never takes a message off the queue.
- Configurable backlog threshold evaluation with dry-run protection.
- Instant Slack alerts detailing queue backlog spikes before consumer exhaustion occurs.
- Centralized execution lineage and audit history inside Kestra.
Who it's for
- Site Reliability Engineers (SREs) and Messaging Platform engineers maintaining RabbitMQ clusters.
- Data Engineers running real-time streaming pipelines who need automated consumer lag alerting.
- System Administrators looking to eliminate opaque custom queue monitoring scripts.
Why orchestrate this with Kestra
Monitoring RabbitMQ queue depth with custom cron scripts or external exporter plugins leaves monitoring isolated from your primary workflow orchestrator. Kestra provides declarative YAML management, built-in secret management, native Slack notification blocks, and execution logging in a single pane of glass. By running queue health audits directly inside Kestra, SREs gain end-to-end visibility and can chain downstream recovery flows seamlessly.
Prerequisites
- A running RabbitMQ broker with the management plugin enabled (the
rabbitmq:*-management images have it), reachable from Kestra.
- A RabbitMQ user with the
management (or monitoring) tag and access to the queue's virtual host.
- A configured Slack Incoming Webhook URL.
Quick start
Import this blueprint into your Kestra instance, configure your AMQP broker connection secrets (RABBITMQ_HOST, RABBITMQ_MANAGEMENT_PORT, etc.), and run an initial execution to read the queue backlog.
How to extend
Extend this flow by chaining container auto-scaling triggers or copying dead-letter payloads into object storage (e.g., AWS S3 or Google Cloud Storage) when queue backlog thresholds are breached.
Secrets
- RABBITMQ_HOST: Hostname or IP address of the RabbitMQ broker.
- RABBITMQ_MANAGEMENT_PORT: Port of the RabbitMQ management HTTP API (e.g. 15672).
- RABBITMQ_USERNAME: Authentication username for AMQP broker access.
- RABBITMQ_PASSWORD: Authentication password for AMQP broker access.
- RABBITMQ_VHOST: Virtual host path (e.g. / or /production).
- SLACK_WEBHOOK: Slack Incoming Webhook URL for alerting.
Note on OSS Secrets: In open-source (OSS) Kestra deployments without an external secret manager, set secrets via environment variables formatted as SECRET_RABBITMQ_HOST, SECRET_RABBITMQ_MANAGEMENT_PORT, SECRET_RABBITMQ_USERNAME, SECRET_RABBITMQ_PASSWORD, SECRET_RABBITMQ_VHOST, and SECRET_SLACK_WEBHOOK.
Inputs
- queue_name (STRING, default: "orders.queue"): Name of the target AMQP queue.
- backlog_threshold (INT, default: 10): Ready message count above which the flow alerts.
- remediation_action (SELECT, default: "ALERT_ONLY", values: ["ALERT_ONLY", "PURGE_EXPIRED"]): Remediation strategy when threshold is breached.
- dry_run (STRING, default: "true"): Safety flag ensuring remediation logic is executed in dry-run mode unless explicitly disabled.
Outputs
- Queue details from the management API, including messages_ready, messages_unacknowledged and consumers: {{ outputs.probe_queue.body }}
Links
Pitfalls
- Port Conflicts: Ensure host port 5672 or 15672 does not conflict with existing local database or broker containers.
- Virtual Host Scoping: AMQP queues are scoped to specific virtual hosts. Verify RABBITMQ_VHOST matches your queue setup.
- Credential Permissions: The management API rejects users without a management tag. Give the configured user the
management or monitoring tag and access to the virtual host; it doesn't need consume permissions.
- Stats Interval: The management API refreshes queue counters every 5 seconds by default, so the depth can lag the broker by a few seconds.
Local Container Setup
To start a local RabbitMQ broker container for testing, execute:
docker run -d --name kestra-rabbitmq -p 5673:5672 -p 15673:15672 -e RABBITMQ_DEFAULT_USER=guest -e RABBITMQ_DEFAULT_PASS=guest rabbitmq:3-management
If Kestra is running inside a Docker network, connect the broker container to Kestra's network:
docker network connect YOUR_KESTRA_NETWORK kestra-rabbitmq
(Run docker network ls to verify your active Kestra network name).