Hazelcast Cluster State & Partition Safety Sentinel

id: hazelcast-cluster-state-and-partition-safety-sentinel
namespace: company.platform

description: "Continuously monitors Hazelcast in-memory data grid cluster state,
  partition safety, and migration queues, alerting on partition desync and
  safely restoring active cluster operational state."

inputs:
  - id: hazelcast_host
    type: STRING
    defaults: "localhost"
    description: "Hostname or IP address of the Hazelcast cluster node."

  - id: hazelcast_port
    type: INT
    defaults: 5701
    description: "HTTP REST Management and Health port for Hazelcast (default: 5701)."

  - id: cluster_name
    type: STRING
    defaults: "dev"
    description: "Name of the Hazelcast cluster being monitored."

  - id: max_migration_queue_threshold
    type: INT
    defaults: 100
    description: "Maximum allowable pending partition migrations before flagging an
      internal rebalance bottleneck."

  - id: auto_restore_active_state
    type: BOOL
    defaults: false
    description: "When enabled and dry_run is false, automatically invokes the
      cluster management API to transition degraded clusters back to ACTIVE
      state."

  - id: dry_run
    type: BOOL
    defaults: true
    description: "Safety gate: when true, audits and emits Slack incident alerts
      without mutating Hazelcast cluster state."

  - id: slack_channel
    type: STRING
    defaults: "#sre-alerts"
    description: "Target Slack channel for Hazelcast partition safety breaches and
      cluster state degradation alerts."

triggers:
  - id: periodic_hazelcast_audit
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "*/15 * * * *"
    disabled: true

tasks:
  - id: probe_cluster_state
    type: io.kestra.plugin.core.http.Request
    description: "Queries Hazelcast Health API to inspect global cluster operational
      state (ACTIVE, FROZEN, or PASSIVE)."
    uri: "http://{{ inputs.hazelcast_host }}:{{ inputs.hazelcast_port
      }}/hazelcast/health/cluster-state"
    method: GET
    options:
      timeout:
        readIdleTimeout: PT15S

  - id: probe_cluster_safety
    type: io.kestra.plugin.core.http.Request
    description: "Queries Hazelcast Health API to verify if all partition backups
      are synchronized across active cluster members."
    uri: "http://{{ inputs.hazelcast_host }}:{{ inputs.hazelcast_port
      }}/hazelcast/health/cluster-safe"
    method: GET
    options:
      timeout:
        readIdleTimeout: PT15S

  - id: probe_migration_queue
    type: io.kestra.plugin.core.http.Request
    description: "Queries Hazelcast Health API to evaluate pending partition
      migration queue depth."
    uri: "http://{{ inputs.hazelcast_host }}:{{ inputs.hazelcast_port
      }}/hazelcast/health/migration-queue-size"
    method: GET
    options:
      timeout:
        readIdleTimeout: PT15S

  - id: classify_cluster_health
    type: io.kestra.plugin.core.flow.Switch
    description: "Evaluates health probe responses to categorize cluster state into
      PARTITION_DATA_SAFETY_BREACH, CLUSTER_STATE_DEGRADED,
      MIGRATION_BACKLOG_DETECTED, or HEALTHY."
    value: >-
      {% if '"clusterSafe":false' in outputs.probe_cluster_safety.body %}
      PARTITION_DATA_SAFETY_BREACH {% elif not ('"clusterState":"ACTIVE"' in
      outputs.probe_cluster_state.body) %} CLUSTER_STATE_DEGRADED {% elif
      (fromJson(outputs.probe_migration_queue.body).migrationQueueSize ?? 0) >
      inputs.max_migration_queue_threshold %} MIGRATION_BACKLOG_DETECTED {% else
      %} HEALTHY {% endif %}
    cases:
      PARTITION_DATA_SAFETY_BREACH:
        - id: alert_partition_safety_slack
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          description: "Dispatches critical incident notification when partition replicas
            are desynchronized and at risk of data loss."
          url: "{{ secret('SLACK_WEBHOOK_URL') }}"
          payload: |
            {
              "channel": "{{ inputs.slack_channel }}",
              "text": ":rotating_light: *Hazelcast Cluster Critical: Partition Safety Breach*",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": ":rotating_light: *Hazelcast Cluster Critical: Partition Safety Breach*\n*Cluster:* `{{ inputs.cluster_name }}` (`http://{{ inputs.hazelcast_host }}:{{ inputs.hazelcast_port }}`)\n*Cluster Safe:* `false`\n*Impact:* In-memory partitions lack full replica quorum. A subsequent node crash will cause irrecoverable data loss."
                  }
                }
              ]
            }

        - id: log_partition_warning
          type: io.kestra.plugin.core.log.Log
          message: "Partition safety breach recorded: cluster {{ inputs.cluster_name }}
            reported clusterSafe=false. Automated node restart suppressed to
            prevent partition destruction."

      CLUSTER_STATE_DEGRADED:
        - id: alert_state_degraded_slack
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          description: "Dispatches warning alert when Hazelcast cluster transitions to
            FROZEN or PASSIVE mode."
          url: "{{ secret('SLACK_WEBHOOK_URL') }}"
          payload: |
            {
              "channel": "{{ inputs.slack_channel }}",
              "text": ":warning: *Hazelcast Cluster Degraded: Non-Active Operational State*",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": ":warning: *Hazelcast Cluster Degraded: Non-Active Operational State*\n*Cluster:* `{{ inputs.cluster_name }}` (`http://{{ inputs.hazelcast_host }}:{{ inputs.hazelcast_port }}`)\n*Current State:* `{{ fromJson(outputs.probe_cluster_state.body).clusterState ?? 'UNKNOWN' }}`\n*Impact:* Cluster writes or membership changes are frozen."
                  }
                }
              ]
            }

        - id: check_state_remediation
          type: io.kestra.plugin.core.flow.If
          description: "Guards automated cluster state transition against invocation
            during dry-run evaluations."
          condition: "{{ inputs.dry_run == false and inputs.auto_restore_active_state ==
            true }}"
          then:
            - id: restore_cluster_state
              type: io.kestra.plugin.core.http.Request
              description: "Invokes Hazelcast Management REST API to transition cluster state
                back to ACTIVE."
              uri: "http://{{ inputs.hazelcast_host }}:{{ inputs.hazelcast_port
                }}/hazelcast/rest/management/cluster/changeState"
              method: POST
              headers:
                Content-Type: "application/x-www-form-urlencoded"
              body: "{{ inputs.cluster_name }}&{{ secret('HAZELCAST_CLUSTER_PASSWORD')
                }}&ACTIVE"
            - id: log_restore_success
              type: io.kestra.plugin.core.log.Log
              message: "Successfully requested state transition to ACTIVE for Hazelcast
                cluster {{ inputs.cluster_name }}."
          else:
            - id: log_dry_run_state
              type: io.kestra.plugin.core.log.Log
              message: "Dry-run mode active or auto_restore_active_state disabled: preserved
                current cluster state without remote REST mutation."

      MIGRATION_BACKLOG_DETECTED:
        - id: alert_migration_backlog_slack
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          description: "Dispatches notice when pending partition migrations exceed cluster
            rebalancing threshold."
          url: "{{ secret('SLACK_WEBHOOK_URL') }}"
          payload: |
            {
              "channel": "{{ inputs.slack_channel }}",
              "text": ":chart_with_upwards_trend: *Hazelcast Cluster Warning: Partition Migration Backlog*",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": ":chart_with_upwards_trend: *Hazelcast Cluster Warning: Partition Migration Backlog*\n*Cluster:* `{{ inputs.cluster_name }}` (`http://{{ inputs.hazelcast_host }}:{{ inputs.hazelcast_port }}`)\n*Pending Migrations:* `{{ fromJson(outputs.probe_migration_queue.body).migrationQueueSize ?? 0 }}` (Threshold: `{{ inputs.max_migration_queue_threshold }}`)\n*Impact:* Cluster partition table is rebalancing under heavy load or following node membership changes."
                  }
                }
              ]
            }

      HEALTHY:
        - id: log_healthy_cluster
          type: io.kestra.plugin.core.log.Log
          description: "Records nominal health status when cluster is active, partitions
            are safe, and migration queue is clear."
          message: "Hazelcast cluster {{ inputs.cluster_name }} healthy on {{
            inputs.hazelcast_host }}:{{ inputs.hazelcast_port }}. State: ACTIVE
            | clusterSafe: true | Migration Queue: {{
            fromJson(outputs.probe_migration_queue.body).migrationQueueSize ?? 0
            }}."

  - id: log_audit_summary
    type: io.kestra.plugin.core.log.Log
    description: "Emits immutable audit log record for operational lineage and
      cluster reliability visibility."
    message: "Hazelcast sentinel audit completed for cluster {{ inputs.cluster_name
      }} at http://{{ inputs.hazelcast_host }}:{{ inputs.hazelcast_port }}.
      Classification: {{ outputs.classify_cluster_health.value | trim }} | Safe:
      {{ fromJson(outputs.probe_cluster_safety.body).clusterSafe ?? false }} |
      Execution: {{ execution.id }}"

errors:
  - id: alert_hazelcast_probe_failure
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    description: "Dispatches out-of-band incident notification if Hazelcast REST
      endpoints are unreachable or timing out."
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {
        "channel": "{{ inputs.slack_channel }}",
        "text": ":rotating_light: *Hazelcast Sentinel Probe Failure*",
        "blocks": [
          {
            "type": "section",
            "text": {
              "type": "mrkdwn",
              "text": ":rotating_light: *Hazelcast Sentinel Probe Failure*\nFailed to probe Hazelcast node at `http://{{ inputs.hazelcast_host }}:{{ inputs.hazelcast_port }}`. The node process may be offline, network route blocked, or REST health endpoints disabled."
            }
          }
        ]
      }

Hazelcast is the leading open-source in-memory data grid (IMDG) and distributed computing platform, widely deployed across financial services, telecommunications, and real-time streaming architectures for low-latency caching and microservice state storage. In production Kubernetes or bare-metal deployments, network latency spikes, node restarts, or heavy GC pauses can trigger partition migration storms or transition the cluster into degraded modes such as PASSIVE or FROZEN. Most critically, when clusterSafe evaluates to false, partition backups are incomplete across member nodes, meaning any subsequent node failure risks permanent in-memory data loss. This blueprint continuously monitors Hazelcast cluster operational state, evaluates partition backup safety, audits migration queue depth, dispatches Slack alerts, and provides guarded self-healing remediation to restore active operational state.

How it works

periodic_hazelcast_audit (io.kestra.plugin.core.trigger.Schedule) initiates automated surveillance every 15 minutes. Shipped disabled so initial manual verification can be performed first.
probe_cluster_state (io.kestra.plugin.core.http.Request) invokes GET /hazelcast/health/cluster-state to verify whether the cluster is ACTIVE, PASSIVE, or FROZEN.
probe_cluster_safety (io.kestra.plugin.core.http.Request) invokes GET /hazelcast/health/cluster-safe to verify that all partition replicas are in sync across members.
probe_migration_queue (io.kestra.plugin.core.http.Request) invokes GET /hazelcast/health/migration-queue-size to inspect pending partition rebalances.
classify_cluster_health (io.kestra.plugin.core.flow.Switch) evaluates probe responses:
- PARTITION_DATA_SAFETY_BREACH: Fired when clusterSafe is false. Dispatches an urgent incident alert to Slack via io.kestra.plugin.slack.notifications.SlackIncomingWebhook warning SREs that partition replicas are desynchronized.
- CLUSTER_STATE_DEGRADED: Fired when the cluster is in PASSIVE or FROZEN mode. Dispatches a Slack notification, and if auto_restore_active_state is enabled with dry_run disabled, executes restore_cluster_state (POST /hazelcast/rest/management/cluster/changeState) to return the cluster to ACTIVE.
- MIGRATION_BACKLOG_DETECTED: Warns the platform team when pending migrations exceed threshold capacity.
- HEALTHY: Confirms nominal cluster state, partition safety, and empty migration queues.
log_audit_summary (io.kestra.plugin.core.log.Log) writes an immutable audit record documenting evaluated cluster state and safety metrics.
Global errors handler captures connection drops, timeouts, or unreachable node instances, immediately routing a probe failure alert to Slack.

What you get

Proactive detection of partition replica desynchronization (clusterSafe=false) before secondary node failures trigger permanent data loss.
Automated detection of cluster freeze or passive states that silently block application writes.
Safe guarded self-healing: dry_run defaults to true to ensure zero unintended cluster mutations during rollout.
Direct Slack notifications containing cluster name, node address, and exact failure metrics for accelerated MTTR.

Who it's for

Site Reliability Engineers and Platform Teams managing production Hazelcast IMDG clusters on Kubernetes or cloud infrastructure.
Distributed Systems Engineers maintaining caching layers, session stores, and real-time computing grids.
Database Administrators overseeing in-memory data integrity and partition replication compliance.

Why orchestrate this with Kestra

Hazelcast provides low-level health REST endpoints but lacks built-in alerting, scheduling, or multi-step self-healing workflows. Relying on custom shell scripts or ad-hoc cron jobs introduces single points of failure, credential exposure, and zero audit lineage. Kestra delivers a declarative, version-controlled YAML blueprint with native scheduling, credential protection through secret masking, conditional branching, and complete execution observability.

Prerequisites

A running Hazelcast cluster member (version 5.0+) accessible over HTTP with REST endpoints enabled. Hazelcast publishes an official open-source Docker image (hazelcast/hazelcast:latest) requiring no commercial license.
An incoming Slack webhook URL for operational notifications.

Secrets

SLACK_WEBHOOK_URL: Slack incoming webhook URL for operational alert notifications.
HAZELCAST_CLUSTER_PASSWORD: Cluster management password required for state transition remediation requests.

Quick start

Start a local Hazelcast test container with health REST endpoints enabled:

docker run -d --name hazelcast-sentinel-test -p 5701:5701 -e HZ_NETWORK_RESTAPI_ENABLED=true -e HZ_NETWORK_RESTAPI_ENDPOINTGROUPS_HEALTHCHECK_ENABLED=true hazelcast/hazelcast:latest

Configure SLACK_WEBHOOK_URL and HAZELCAST_CLUSTER_PASSWORD secrets in your Kestra namespace.
Import this blueprint into Kestra and trigger an execution with default inputs (dry_run: true).
Verify that log_healthy_cluster logs nominal cluster state and partition safety.

Test state degradation detection by temporarily changing cluster state:

curl -X POST http://localhost:5701/hazelcast/rest/management/cluster/changeState -d "dev&password&FROZEN"

Re-run the flow to observe alert_state_degraded_slack firing.
Enable periodic_hazelcast_audit to activate recurring 15-minute surveillance.

How to extend

Add Prometheus metric exports via io.kestra.plugin.prometheus to track partition migration queue trends and member counts over time.
Integrate with PagerDuty via io.kestra.plugin.pagerduty.notifications.PagerDutyAlert for high-severity partition safety violations.
Add multi-member health sweeps iterating through all cluster node IPs using io.kestra.plugin.core.flow.EachParallel.

Hazelcast Cluster State & Partition Safety Sentinel

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Hazelcast Cluster State & Partition Safety Sentinel

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

AI-Powered Incident Triage & Automated Response

Autonomous Multi-Agent Software Engineering and Self-Healing Pipeline

AI Security Incident Threat Advisor and MITRE ATT&CK Containment Generator

AI Kubernetes Workload Hardening and Security Policy Advisor

Autonomous React PR Reviewer with Playwright Visual Regression Diff

Autonomous React Component Generator with Deep Learning Visual QA

New to Kestra?