Schedule icon
Script icon
If icon
Create icon
SlackIncomingWebhook icon
Log icon

RabbitMQ Dynamic Shovel Failure and Cross-Cluster Sentinel

Monitor RabbitMQ dynamic shovels and cross-cluster replication with Kestra. Query management endpoints, create GitHub incident issues, and alert Slack.

Categories
CloudInfrastructure

RabbitMQ shovels reliably forward messages between geographically distributed broker clusters, isolated VPC networks, or distinct virtual hosts. However, transient network partitions, credential expirations, or target broker resource exhaustion can silently place shovels into a terminated or stopped state, causing massive queue backlog accumulation on source brokers. This blueprint polls RabbitMQ's Management API periodically, detects non-running dynamic shovels, logs a prioritized GitHub investigation ticket, and alerts on-call messaging SREs via Slack.

How it works

  1. check_shovel_status_periodic (io.kestra.plugin.core.trigger.Schedule) initiates the audit every 15 minutes (shipped disabled for safe verification).
  2. probe_shovel_states (io.kestra.plugin.scripts.python.Script) queries RabbitMQ's Management REST API (GET /api/shovels/{vhost}) and identifies any shovel whose status is not running.
  3. branch_on_shovel_failures (io.kestra.plugin.core.flow.If) evaluates whether degraded shovels exist (has_failed_shovels == true):
    • then: Automatically files a GitHub tracking issue via create_shovel_incident_issue (io.kestra.plugin.github.issues.Create) and alerts the Slack channel via alert_messaging_slack (io.kestra.plugin.slack.notifications.SlackIncomingWebhook).
    • else: Emits an informational log entry (log_shovels_healthy) confirming normal message movement.
  4. log_audit_complete (io.kestra.plugin.core.log.Log) records the audit outcome.
  5. The errors block alerts Slack if management API authentication fails or the host is unreachable.

What you get

  • Automated, continuous visibility into cross-cluster message replication pipelines.
  • Zero silent queue accumulation on upstream producer clusters.
  • Consolidated GitHub incident tickets with broken shovel names and current states.
  • Real-time Slack notifications with direct ticket hyperlinks.

Who it's for

  • Platform Engineers and Messaging SREs maintaining RabbitMQ architectures.
  • Distributed systems developers relying on cross-region message replication.
  • Cloud Infrastructure architects governing event-driven message backbones.

Why orchestrate this with Kestra

Kestra coordinates external API polling, JSON parsing, conditional logic, GitHub issue tracking, and team alerting in a single declarative workflow. It manages broker credentials securely, provides auditability for every check, and eliminates the need for separate standalone monitoring daemons.

Prerequisites

  • A RabbitMQ cluster with the rabbitmq_shovel_management plugin enabled.
  • A RabbitMQ user with monitoring or administrator permissions on the management API.
  • A GitHub Personal Access Token with repository issue creation permissions.
  • A Slack incoming webhook URL for notifications.

Secrets

  • RABBITMQ_USER, RABBITMQ_PASSWORD: Authentication credentials for the RabbitMQ Management API.
  • GITHUB_TOKEN: GitHub token used to open messaging incident issues.
  • SLACK_WEBHOOK: Incoming Webhook URL for alerting.

Inputs

  • rabbitmq_host (STRING, default rabbitmq.internal.company.com:15672): Management API hostname and port.
  • vhost (STRING, default %2F): URL-encoded virtual host name hosting the dynamic shovels.
  • github_repository (STRING, default my-org/messaging-platform): Target GitHub repository for logging incident issues.

Outputs

  • outputs.probe_shovel_states.vars.total_shovels: Total number of dynamic shovels discovered.
  • outputs.probe_shovel_states.vars.failed_count: Number of shovels in a degraded or stopped state.
  • outputs.probe_shovel_states.vars.failed_shovels: List of failed shovel objects (name, vhost, state, timestamp).

Quick start

  1. Configure RABBITMQ_USER, RABBITMQ_PASSWORD, GITHUB_TOKEN, and SLACK_WEBHOOK in your Kestra namespace.
  2. Import this flow and set rabbitmq_host and vhost to match your cluster.
  3. Run the flow manually to verify shovel status extraction and Slack notification delivery.
  4. Enable the check_shovel_status_periodic trigger for continuous message replication monitoring.

How to extend

  • Add an automated shovel restart API call (POST /api/shovels/{vhost}/{name}/restart) to attempt self-healing.
  • Pull unacknowledged message counts from upstream queues alongside shovel states.
  • Route critical alerts to PagerDuty or Opsgenie if high-priority shovels remain stopped for over 30 minutes.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.