API Evolution Watchtower
Monitor third-party OpenAPI specifications for consumer-impacting contract changes and govern accepted revisions with persistent PASS/WARN/BLOCK decisions.
Audit RabbitMQ AMQP message TTL expiration and dead-letter backlog with Kestra. Detect head-of-line blocking and dispatch automated alerts.
In RabbitMQ and AMQP architectures, messages published with per-message Time-To-Live (TTL) or queues configured with message expiration are intended to be processed or dropped within a deterministic timeframe. However, RabbitMQ only evaluates per-message TTL when a message reaches the head of the queue. If an unexpired or stuck message blocks the head, newer expired messages remain accumulated behind it, silently consuming broker memory. Furthermore, when messages finally expire and route to a Dead-Letter Queue (DLQ), consumer drop-offs can cause unmonitored DLQ backlog spikes.
This blueprint provides an automated Day 2 Operations sentinel flow that reads DLQ backlog depth from the RabbitMQ management API on a scheduled cadence, without consuming any message. It classifies queue health against configurable warning thresholds, alerts on-call engineers via Slack upon backlog accumulation, and logs actionable diagnostic remediation telemetry.
schedule (io.kestra.plugin.core.trigger.Schedule) initiates the audit cycle every 6 hours (disabled by default for safe initial setup).probe_dlq_backlog (io.kestra.plugin.core.http.Request) reads dlq_queue_name from the RabbitMQ management API (GET /api/queues/{vhost}/{queue}). It only reads the queue's counters, so the dead-lettered messages stay on the broker for inspection or replay.evaluate_threshold (io.kestra.plugin.core.flow.Switch) evaluates whether the DLQ's messages_ready exceeds inputs.expired_threshold.HEALTHY), log_healthy_status records the successful pass.BREACH), alert_threshold_breach dispatches an operational alert to Slack using io.kestra.plugin.slack.notifications.SlackIncomingWebhook, and log_remediation_action logs the chosen remediation mode.log_audit_completion writes a final audit recap for execution tracking and observability.errors block intercepts broker connectivity failures or authentication errors and notifies Slack.Monitoring RabbitMQ TTL and DLQ accumulation with standalone cron scripts requires managing custom credentials, error handling, retry logic, and webhook clients in isolated codebases. Kestra provides a declarative workflow engine with native AMQP consumption, centralized secret management, conditional branching, execution history, and integrated Slack alerting without writing or maintaining custom polling daemons.
rabbitmq:*-management images have it), reachable from Kestra.x-dead-letter-exchange) and DLQ.To run RabbitMQ locally for development and testing while avoiding host port 8080 conflicts:
docker run -d --name rabbitmq-dev -p 5672:5672 -p 15672:15672 -e RABBITMQ_DEFAULT_USER=guest -e RABBITMQ_DEFAULT_PASS=guest rabbitmq:3-management
If Kestra is running inside a Docker network, connect the RabbitMQ container to Kestra network:
docker network ls
docker network connect YOUR_KESTRA_NETWORK rabbitmq-dev
RABBITMQ_HOST: Hostname or container name of the RabbitMQ broker (e.g. rabbitmq-dev or localhost).RABBITMQ_MANAGEMENT_PORT: Management HTTP API port (e.g. 15672).RABBITMQ_USERNAME: RabbitMQ user with the management or monitoring tag.RABBITMQ_PASSWORD: Password for the RabbitMQ user.RABBITMQ_VHOST: Virtual host name (default /).SLACK_WEBHOOK: Incoming Webhook URL for Slack alerting.Note for Kestra Open-Source (OSS): Secrets are provided as base64-encoded environment variables formatted as
SECRET_<NAME>on the Kestra server or worker environment.
| Name | Type | Default | Allowed Values | Description |
|---|---|---|---|---|
queue_name |
STRING | "orders.queue" |
Any valid queue name | Primary AMQP queue configured with TTL |
dlq_queue_name |
STRING | "orders.dlq" |
Any valid queue name | Dead-letter queue capturing expired messages |
expired_threshold |
INT | 5 |
Positive integer | Threshold of expired messages triggering alert |
remediation_mode |
SELECT | "ALERT_ONLY" |
"ALERT_ONLY", "QUARANTINE_LOG" |
Operational remediation action mode |
dry_run |
SELECT | "true" |
"true", "false" |
Dry run simulation flag |
{{ outputs.probe_dlq_backlog.body }}: The DLQ's details from the management API, including messages_ready, messages_unacknowledged and consumers.RABBITMQ_HOST, RABBITMQ_MANAGEMENT_PORT, RABBITMQ_USERNAME, RABBITMQ_PASSWORD, RABBITMQ_VHOST, and SLACK_WEBHOOK in your Kestra namespace.dry_run: "true" to verify connectivity and the DLQ depth read.schedule trigger to run automated recurring audits.io.kestra.plugin.amqp.CreateQueue or QueueBind.BREACH branch.management or monitoring tag and access to the virtual host; it doesn't need consume permissions.