Cassandra Thread Pool Exhaustion Sentinel

id: cassandra-thread-pool-exhaustion-sentinel
namespace: company.database

description: |
  Monitors Apache Cassandra 4.0+ thread pools via virtual tables on a schedule, detects
  thread pool saturation and blocked tasks before query latency spikes, routes remediation
  telemetry, and dispatches incident alerts to Slack.

inputs:
  - id: contact_points
    type: STRING
    displayName: Cassandra Node Hostname
    defaults: "cassandra"
    description: Hostname or IP address of the Cassandra node contact point
      reachable from the Kestra worker.

  - id: contact_port
    type: INT
    displayName: Cassandra CQL Port
    defaults: 9042
    description: Native CQL protocol port (defaults to 9042).

  - id: local_datacenter
    type: STRING
    displayName: Local Datacenter Name
    defaults: "datacenter1"
    description: Cassandra datacenter name configured in cassandra.yaml (e.g.,
      datacenter1 or us-east-1).

  - id: max_pending_threshold
    type: INT
    displayName: Max Pending Tasks Threshold
    defaults: 50
    description: Maximum pending queue depth permitted before flagging a thread pool
      as saturated.

  - id: slack_webhook
    type: STRING
    displayName: Slack Webhook URL
    defaults: "https://hooks.slack.com/services/PLACEHOLDER/WEBHOOK/URL"
    description: Slack Incoming Webhook URL to deliver operational notifications.

tasks:
  - id: probe_thread_pools
    type: io.kestra.plugin.cassandra.standard.Query
    description: Query Cassandra 4.0+ system_views.thread_pools virtual table to
      retrieve active, pending, and blocked task counters across all stages.
    session:
      endpoints:
        - hostname: "{{ inputs.contact_points }}"
          port: "{{ inputs.contact_port }}"
      localDatacenter: "{{ inputs.local_datacenter }}"
    cql: "SELECT name, active_tasks, pending_tasks, blocked_tasks FROM
      system_views.thread_pools;"
    fetchType: FETCH
    allowFailure: true

  - id: classify_thread_pools
    type: io.kestra.plugin.core.debug.Return
    description: Classify cluster node saturation into HEALTHY, EXHAUSTED, or
      UNREACHABLE based on virtual table metrics.
    format: "{{ outputs.probe_thread_pools.rows is defined ?
      ((outputs.probe_thread_pools.rows | jq('[.[] | select(.blocked_tasks > 0
      or .pending_tasks > ' ~ inputs.max_pending_threshold ~ ')] | length') |
      first | default(0)) > 0 ? 'EXHAUSTED' : 'HEALTHY') : 'UNREACHABLE' }}"

  - id: route_by_saturation
    type: io.kestra.plugin.core.flow.Switch
    description: Branch workflow execution based on evaluated thread pool health state.
    value: "{{ outputs.classify_thread_pools.value }}"
    cases:
      HEALTHY:
        - id: log_healthy_pools
          type: io.kestra.plugin.core.log.Log
          description: Log healthy status when all thread pools report zero blocked tasks
            and acceptable pending queues.
          message: "Cassandra node {{ inputs.contact_points }} (DC: {{
            inputs.local_datacenter }}) is healthy. Inspected {{
            outputs.probe_thread_pools.size }} thread pools with 0 blocked
            tasks."

      EXHAUSTED:
        - id: alert_thread_pool_exhaustion
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          description: Dispatch urgent incident notification to Slack upon thread pool
            saturation.
          url: "{{ inputs.slack_webhook }}"
          payload: |
            {
              "channel": "#cassandra-alerts",
              "username": "Cassandra-Sentinel",
              "text": ":warning: *Cassandra Thread Pool Exhaustion Alert*\n*Node*: `{{ inputs.contact_points }}` (DC: `{{ inputs.local_datacenter }}`)\n*Details*: Thread pool saturation detected. One or more stages have blocked tasks or exceed {{ inputs.max_pending_threshold }} pending tasks.\n*Saturated Pools*: `{{ outputs.probe_thread_pools.rows | jq('[.[] | select(.blocked_tasks > 0 or .pending_tasks > ' ~ inputs.max_pending_threshold ~ ')]') | first }}`"
            }

        - id: log_exhaustion_telemetry
          type: io.kestra.plugin.core.log.Log
          description: Record saturation diagnostics and remediation runbook
            recommendations for SRE triage.
          message: "CRITICAL: Cassandra thread pool saturation on {{ inputs.contact_points
            }}. Overloaded pools: {{ outputs.probe_thread_pools.rows | jq('[.[]
            | select(.blocked_tasks > 0 or .pending_tasks > ' ~
            inputs.max_pending_threshold ~ ')]') | first }}. Immediate
            remediation advice: review disk I/O latency, JVM GC pause times, and
            compaction backlog."

      UNREACHABLE:
        - id: alert_unreachable_node
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          description: Dispatch connection failure warning to Slack when the Cassandra
            node cannot be queried.
          url: "{{ inputs.slack_webhook }}"
          payload: |
            {
              "channel": "#cassandra-alerts",
              "username": "Cassandra-Sentinel",
              "text": ":rotating_light: *Cassandra Node Unreachable*\n*Node*: `{{ inputs.contact_points }}`:{{ inputs.contact_port }} (DC: `{{ inputs.local_datacenter }}`)\n*Execution ID*: `{{ execution.id }}`\n*Alert*: Sentinel was unable to connect to CQL native transport or retrieve `system_views.thread_pools`."
            }

        - id: log_unreachable_warning
          type: io.kestra.plugin.core.log.Log
          description: Log cluster connectivity warning.
          message: "WARNING: Unable to reach Cassandra node {{ inputs.contact_points }}:{{
            inputs.contact_port }} or retrieve system_views.thread_pools."

  - id: audit_sentinel_run
    type: io.kestra.plugin.core.log.Log
    description: Emit structured execution recap and audit log for telemetry compliance.
    message: "Cassandra thread pool sentinel finished for {{ inputs.contact_points
      }} in {{ inputs.local_datacenter }}. Inspected {{
      outputs.probe_thread_pools.size }} thread pools. Evaluation: {{
      outputs.classify_thread_pools.value }}."

errors:
  - id: handle_execution_failure
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    description: Capture unexpected flow execution errors and notify the on-call channel.
    url: "{{ inputs.slack_webhook }}"
    payload: |
      {
        "channel": "#cassandra-alerts",
        "username": "Cassandra-Sentinel",
        "text": ":fire: *Cassandra Sentinel Workflow Error*\nWorkflow `{{ flow.id }}` encountered an unexpected runtime failure during execution `{{ execution.id }}`."
      }

triggers:
  - id: periodic_thread_pool_audit
    type: io.kestra.plugin.core.trigger.Schedule
    description: Scheduled execution to inspect Cassandra thread pool metrics every
      5 minutes.
    cron: "*/5 * * * *"
    disabled: true

Apache Cassandra relies on staged event-driven architecture (SEDA) thread pools (such as MutationStage, ReadStage, CompactionExecutor, and Native-Transport-Requests) to handle concurrency, read/write I/O, and background maintenance. Under sudden write bursts, slow storage, or JVM garbage collection pauses, task queues fill up and tasks become blocked, leading to client request drops, elevated P99 latency, and cluster degradation. Cassandra 4.0 introduces the system_views.thread_pools virtual table, exposing in-memory thread pool metrics directly over native CQL without requiring external JMX exporters. This blueprint audits Cassandra thread pools on a schedule, detects backlog accumulation or blocked tasks, logs diagnostic telemetry, and dispatches real-time incident notifications to Slack.

How it works

periodic_thread_pool_audit (io.kestra.plugin.core.trigger.Schedule) initiates scheduled monitoring every 5 minutes (shipped disabled for safe onboarding).
probe_thread_pools (io.kestra.plugin.cassandra.standard.Query) queries the native CQL virtual table system_views.thread_pools to extract name, active_tasks, pending_tasks, and blocked_tasks with fetchType: FETCH and allowFailure: true.
classify_thread_pools (io.kestra.plugin.core.debug.Return) evaluates virtual table outputs using Pebble expressions and jq filters, categorizing the node state as HEALTHY (zero blocked tasks and pending queues within limit), EXHAUSTED (any stage with blocked tasks > 0 or pending tasks > max_pending_threshold), or UNREACHABLE (connection failure or empty output).
route_by_saturation (io.kestra.plugin.core.flow.Switch) routes downstream actions based on the evaluated state:
- HEALTHY: logs cluster stability and pool counts via log_healthy_pools.
- EXHAUSTED: sends an incident alert to Slack with the offending stage details via alert_thread_pool_exhaustion and emits remediation runbook instructions through log_exhaustion_telemetry.
- UNREACHABLE: dispatches a node connectivity alert via alert_unreachable_node and records a warning log.
audit_sentinel_run (io.kestra.plugin.core.log.Log) produces a structured execution recap tracking the inspected host, datacenter, pool count, and classification verdict.
handle_execution_failure in the flow-level errors handler catches unhandled workflow execution errors and notifies Slack.

What you get

Proactive detection of Cassandra thread pool bottlenecks before clients experience read/write timeouts.
Zero JMX dependency by leveraging Cassandra 4.0+ native CQL virtual tables (system_views.thread_pools).
Rich contextual incident alerts in Slack detailing the exact saturated thread pool names and queue depths.
Automated triage runbook guidance in logs pointing engineers directly to root causes (disk I/O, GC pauses, compaction lag).
Safe-by-default execution with parameterized thresholds and an initially disabled schedule.

Who it's for

Database Reliability Engineers (DBREs) managing distributed Cassandra clusters.
Platform and Site Reliability Engineers (SREs) operating mission-critical low-latency data pipelines.
DevOps engineers seeking lightweight database health checks without deploying heavy monitoring agents.

Why orchestrate this with Kestra

Operating Cassandra at scale requires continuous visibility into internal stage queues, but traditional monitoring often depends on fragile JMX exporters, Jolokia sidecars, or complex external scraping stacks that require daemon maintenance. Orchestrating thread pool health checks directly through Kestra provides:

Declarative, version-controlled Cassandra health automation running as standard workflows.
First-class Pebble and jq transformations to classify multi-stage metrics into actionable operational states without maintaining glue code.
Centralized secrets management ensuring database credentials and Slack webhooks are securely handled.
Native execution history and full execution audit trails across every scheduled check.

Prerequisites

A running Apache Cassandra 4.0+ cluster or open-source Docker container (cassandra:4.1).
Network connectivity from the Kestra worker to the Cassandra native CQL transport port (default 9042).
A Slack Incoming Webhook URL with posting permissions to the designated operations channel.

Secrets

SLACK_WEBHOOK_URL: Slack Incoming Webhook URL used by SlackIncomingWebhook to send incident and probe failure notifications.
CASSANDRA_USERNAME: (Optional) Administrative CQL username if authentication is enabled on the cluster.
CASSANDRA_PASSWORD: (Optional) Administrative CQL password corresponding to CASSANDRA_USERNAME.

Quick start

Store your Slack Incoming Webhook URL in Kestra Secrets under the key SLACK_WEBHOOK_URL (or provide it via the slack_webhook flow input).
If your Cassandra cluster enforces password authentication (PasswordAuthenticator), configure CASSANDRA_USERNAME and CASSANDRA_PASSWORD in Kestra Secrets and add credentials to session.
Execute the flow manually from the Kestra UI to verify successful connection and metric retrieval against your target node.
Review execution logs to confirm all thread pools are reported with zero blocked tasks.
Enable the periodic_thread_pool_audit trigger to start continuous 5-minute automated sentinel monitoring.

How to extend

Auto-remediation with nodetool: Add an SSH or container execution task that runs nodetool setconcurrency or adjusts dynamic compaction throughput when specific thread pools spike.
Multi-node cluster scanning: Wrap the probe task in a ForEach loop iterating over all coordinator and replica node IPs in the cluster.
PagerDuty integration: Pair the Slack notification with io.kestra.plugin.pagerduty.Incident for critical EXHAUSTED alerts occurring during peak business hours.
Metrics export to Prometheus: Forward thread pool metrics to a Pushgateway or metrics sink for historical trend analysis.

Cassandra Thread Pool Exhaustion and Blocked Tasks Sentinel

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Cassandra Thread Pool Exhaustion and Blocked Tasks Sentinel

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

More Data Blueprints

New to Kestra?