Schedule icon
Script icon
If icon
Create icon
SlackIncomingWebhook icon
Log icon

Redis Cluster State and Hash Slot Failover Sentinel

Monitor distributed Redis Cluster health and hash slot coverage with Kestra. Query CLUSTER INFO, open GitHub incident issues, and alert Slack.

Categories
CloudInfrastructure

Enterprise distributed caching architectures rely on Redis Cluster to partition data across multiple master and replica nodes using 16,384 hash slots. When a master node fails and failover encounters network partition delays, or if hash slots become unassigned, the entire cluster transitions into a cluster_state: fail condition, rejecting client read and write operations. This blueprint connects to a Redis Cluster seed node on a regular cadence, queries CLUSTER INFO, detects failing hash slots or degraded cluster states, logs an automated GitHub incident ticket, and alerts on-call infrastructure engineers on Slack.

How it works

  1. periodic_cluster_health_check (io.kestra.plugin.core.trigger.Schedule) initiates the audit every 5 minutes (shipped disabled for safe verification).
  2. probe_redis_cluster_info (io.kestra.plugin.scripts.python.Script) connects to the target Redis node and parses CLUSTER INFO fields, reading cluster_state, cluster_slots_assigned, cluster_slots_ok, and cluster_slots_fail.
  3. branch_on_cluster_health (io.kestra.plugin.core.flow.If) evaluates whether the cluster state is degraded or slots are failing (is_degraded == true):
    • then: Automatically files a GitHub tracking issue via create_cache_incident_issue (io.kestra.plugin.github.issues.Create) and alerts the Slack channel via alert_cache_slack (io.kestra.plugin.slack.notifications.SlackIncomingWebhook).
    • else: Emits an informational log entry (log_cluster_healthy) confirming normal operational health.
  4. log_audit_complete (io.kestra.plugin.core.log.Log) records the final health summary.
  5. The errors block alerts Slack if network connection times out or authentication fails.

What you get

  • Automated, scheduled auditing of distributed Redis Cluster topologies.
  • Zero delayed detection when master node failures cause slot unassignment.
  • Consolidated GitHub incident tickets with healthy vs failing slot counts and node tallies.
  • Real-time Slack notifications with direct ticket hyperlinks.

Who it's for

  • Platform Engineers and Database SREs operating production Redis Cluster deployments.
  • Distributed systems developers relying on high-availability in-memory caching.
  • Site Reliability Engineers maintaining service-level objectives for caching layers.

Why orchestrate this with Kestra

Kestra coordinates socket-level Redis communication, command output parsing, conditional escalation, and chat alerting inside a unified declarative workflow. It manages cluster passwords securely, provides complete execution history, and eliminates the need for separate standalone monitoring daemons.

Prerequisites

  • A running Redis Cluster deployment (OSS Redis, AWS ElastiCache, or Redis Enterprise).
  • Network connectivity from the Kestra worker to the Redis host and port.
  • A GitHub Personal Access Token with repository issue creation permissions.
  • A Slack incoming webhook URL for notifications.

Secrets

  • REDIS_PASSWORD: Authentication password for Redis Cluster nodes.
  • GITHUB_TOKEN: GitHub token used to open incident issues.
  • SLACK_WEBHOOK: Incoming Webhook URL for infrastructure alerts.

Inputs

  • redis_host (STRING, default redis-cluster.internal.company.com): Hostname or IP of a Redis Cluster seed node.
  • redis_port (INT, default 6379): Redis listening port.
  • target_repository (STRING, default my-org/cache-infrastructure): Target GitHub repository for logging incident issues.

Outputs

  • outputs.probe_redis_cluster_info.vars.cluster_state: Current cluster status (ok or fail).
  • outputs.probe_redis_cluster_info.vars.slots_ok: Number of operational hash slots.
  • outputs.probe_redis_cluster_info.vars.slots_fail: Number of failing hash slots.

Quick start

  1. Configure REDIS_PASSWORD, GITHUB_TOKEN, and SLACK_WEBHOOK in your Kestra namespace.
  2. Import this flow and set redis_host and redis_port to match your cluster endpoint.
  3. Run the flow manually to verify cluster information extraction and Slack notification delivery.
  4. Enable the periodic_cluster_health_check trigger for continuous topology monitoring.

How to extend

  • Add an automated CLUSTER FAILOVER execution task to trigger failover on a designated replica.
  • Interrogate individual node roles via CLUSTER NODES to identify specific failing master IPs.
  • Route urgent alerts to PagerDuty or Opsgenie when slots_fail is greater than zero.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.