Harbor Registry Garbage Collection Execution Stall Sentinel
Audits Harbor enterprise container registry telemetry to detect stalled garbage collection jobs, lock freezes, and storage bloat.
Monitor RabbitMQ dynamic shovels and cross-cluster replication with Kestra. Query management endpoints, create GitHub incident issues, and alert Slack.
RabbitMQ shovels reliably forward messages between geographically distributed broker clusters, isolated VPC networks, or distinct virtual hosts. However, transient network partitions, credential expirations, or target broker resource exhaustion can silently place shovels into a terminated or stopped state, causing massive queue backlog accumulation on source brokers. This blueprint polls RabbitMQ's Management API periodically, detects non-running dynamic shovels, logs a prioritized GitHub investigation ticket, and alerts on-call messaging SREs via Slack.
check_shovel_status_periodic (io.kestra.plugin.core.trigger.Schedule) initiates the audit every 15 minutes (shipped disabled for safe verification).probe_shovel_states (io.kestra.plugin.scripts.python.Script) queries RabbitMQ's Management REST API (GET /api/shovels/{vhost}) and identifies any shovel whose status is not running.branch_on_shovel_failures (io.kestra.plugin.core.flow.If) evaluates whether degraded shovels exist (has_failed_shovels == true):then: Automatically files a GitHub tracking issue via create_shovel_incident_issue (io.kestra.plugin.github.issues.Create) and alerts the Slack channel via alert_messaging_slack (io.kestra.plugin.slack.notifications.SlackIncomingWebhook).else: Emits an informational log entry (log_shovels_healthy) confirming normal message movement.log_audit_complete (io.kestra.plugin.core.log.Log) records the audit outcome.errors block alerts Slack if management API authentication fails or the host is unreachable.Kestra coordinates external API polling, JSON parsing, conditional logic, GitHub issue tracking, and team alerting in a single declarative workflow. It manages broker credentials securely, provides auditability for every check, and eliminates the need for separate standalone monitoring daemons.
rabbitmq_shovel_management plugin enabled.RABBITMQ_USER, RABBITMQ_PASSWORD: Authentication credentials for the RabbitMQ Management API.GITHUB_TOKEN: GitHub token used to open messaging incident issues.SLACK_WEBHOOK: Incoming Webhook URL for alerting.rabbitmq_host (STRING, default rabbitmq.internal.company.com:15672): Management API hostname and port.vhost (STRING, default %2F): URL-encoded virtual host name hosting the dynamic shovels.github_repository (STRING, default my-org/messaging-platform): Target GitHub repository for logging incident issues.outputs.probe_shovel_states.vars.total_shovels: Total number of dynamic shovels discovered.outputs.probe_shovel_states.vars.failed_count: Number of shovels in a degraded or stopped state.outputs.probe_shovel_states.vars.failed_shovels: List of failed shovel objects (name, vhost, state, timestamp).RABBITMQ_USER, RABBITMQ_PASSWORD, GITHUB_TOKEN, and SLACK_WEBHOOK in your Kestra namespace.rabbitmq_host and vhost to match your cluster.check_shovel_status_periodic trigger for continuous message replication monitoring.POST /api/shovels/{vhost}/{name}/restart) to attempt self-healing.