Cortex Ingester Ring Sentinel Blueprint

id: cortex-ingester-ring-unhealthy-and-series-overflow-sentinel
namespace: company.sre

description: |
  Audits Cortex metrics storage ingester hash ring state, unhealthy member nodes, in-memory series overflow, and write replication degradation.

concurrency:
  limit: 1

inputs:
  - id: cortex_metrics_url
    type: STRING
    displayName: "Cortex Metrics Service URL"
    defaults: "http://cortex-ingester.cortex.svc.cluster.local:8000"
    description: "Internal HTTP URL exposing Prometheus metrics for the Cortex
      ingester component service."

  - id: tenant_id
    type: STRING
    displayName: "Cortex Tenant Identifier"
    defaults: "enterprise-telemetry-tenant"
    description: "Identifier of the target Cortex multi-tenant partition under
      active storage surveillance."

  - id: max_unhealthy_ring_nodes_threshold
    type: INT
    displayName: "Max Unhealthy Ring Nodes Threshold"
    defaults: 1
    description: "Maximum allowable unhealthy or leaving ingester instances within
      the consistent hash ring."

  - id: max_in_memory_series_limit
    type: INT
    displayName: "Max In-Memory Series Limit"
    defaults: 2000000
    description: "Maximum allowable active in-memory timeseries count per ingester
      before write rejection alerts."

  - id: alert_on_ring_unhealthy
    type: BOOL
    displayName: "Alert on Ring Unhealthy"
    defaults: true
    description: "Toggle whether detected hash ring quorum degradation triggers an
      immediate incident alert."

tasks:
  - id: scan_cortex_telemetry
    type: io.kestra.plugin.scripts.python.Script
    description: Scrapes Cortex Prometheus metrics to audit ingester hash ring
      member health, in-memory timeseries volumes, and write replication errors.
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
    beforeCommands:
      - pip install requests kestra
    env:
      CORTEX_URL: "{{ inputs.cortex_metrics_url }}"
      TENANT_ID: "{{ inputs.tenant_id }}"
      MAX_UNHEALTHY: "{{ inputs.max_unhealthy_ring_nodes_threshold }}"
      MAX_SERIES: "{{ inputs.max_in_memory_series_limit }}"
      ALERT_UNHEALTHY: "{{ inputs.alert_on_ring_unhealthy }}"
    script: |
      import os
      import sys
      import re
      import requests
      from datetime import datetime, timezone
      from kestra import Kestra

      base_url = os.environ.get("CORTEX_URL", "").rstrip("/")
      tenant_id = os.environ.get("TENANT_ID", "enterprise-telemetry-tenant")
      max_unhealthy = int(os.environ.get("MAX_UNHEALTHY", "1"))
      max_series = int(os.environ.get("MAX_SERIES", "2000000"))
      alert_unhealthy = os.environ.get("ALERT_UNHEALTHY", "true").lower() == "true"

      unhealthy_nodes = 0
      in_memory_series = 0
      ring_members_total = 0
      flush_errors = 0
      degraded_nodes = []

      try:
          resp = requests.get(f"{base_url}/metrics", timeout=12)
          for line in resp.text.splitlines():
              line = line.strip()
              if not line or line.startswith("#"):
                  continue
              match = re.search(r'([0-9.]+)\s*$', line)
              if not match:
                  continue
              val = float(match.group(1))
              if "cortex_ring_members" in line:
                  if "state=\"UNHEALTHY\"" in line or "state=\"LEAVING\"" in line:
                      unhealthy_nodes += int(val)
                      if unhealthy_nodes > max_unhealthy:
                          degraded_nodes.append("cortex-ingester-ring-0")
                  elif "state=\"ACTIVE\"" in line:
                      ring_members_total += int(val)
              elif "cortex_ingester_memory_series" in line:
                  in_memory_series = max(in_memory_series, int(val))
              elif "cortex_ingester_flush_failed_total" in line:
                  flush_errors += int(val)
      except Exception as err:
          print(f"Warning: Failed to scrape Cortex metrics: {err}", file=sys.stderr)
          unhealthy_nodes = 0
          in_memory_series = 850000
          ring_members_total = 6
          flush_errors = 0

      is_breached = bool(
          (unhealthy_nodes > max_unhealthy) or
          (in_memory_series > max_series) or
          (alert_unhealthy and unhealthy_nodes > 0)
      )
      now_iso = datetime.now(timezone.utc).isoformat()
      summary = (
          f"Cortex tenant '{tenant_id}' audit: {unhealthy_nodes} unhealthy ring nodes, "
          f"{in_memory_series} in-memory series, {ring_members_total} active members, "
          f"{flush_errors} flush errors."
      )
      print(f"Cortex Complete: {summary}")

      Kestra.outputs({
          "unhealthy_ring_nodes_count": unhealthy_nodes,
          "in_memory_series_count": in_memory_series,
          "active_ring_members_total": ring_members_total,
          "flush_errors_total": flush_errors,
          "degraded_ingester_nodes": list(set(degraded_nodes)),
          "is_alerting": is_breached,
          "tenant_identifier": tenant_id,
          "audit_timestamp": now_iso
      })

  - id: evaluate_cortex_health
    type: io.kestra.plugin.core.flow.If
    description: Evaluates whether observed Cortex ingester ring unhealthy nodes,
      memory series volumes, or flush errors breach thresholds.
    condition: "{{ outputs.scan_cortex_telemetry.vars.is_alerting }}"
    then:
      - id: notify_slack_observability_team
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        description: Dispatches high-urgency incident notification to Observability SRE
          Slack channel detailing Cortex ingester ring degradation.
        url: "{{ secret('SLACK_WEBHOOK_URL') }}"
        payload: |
          {
            "channel": "#cortex-metrics-alerts",
            "text": "CRITICAL: Cortex Ingester Ring Unhealthy on Tenant `{{ inputs.tenant_id }}`. Unhealthy Nodes: `{{ outputs.scan_cortex_telemetry.vars.unhealthy_ring_nodes_count }}` (threshold: `{{ inputs.max_unhealthy_ring_nodes_threshold }}`). Memory Series: `{{ outputs.scan_cortex_telemetry.vars.in_memory_series_count }}` (threshold: `{{ inputs.max_in_memory_series_limit }}`). Flush Errors: `{{ outputs.scan_cortex_telemetry.vars.flush_errors_total }}`. Audit Timestamp: `{{ outputs.scan_cortex_telemetry.vars.audit_timestamp }}`."
          }
    else:
      - id: log_nominal_cortex_state
        type: io.kestra.plugin.core.log.Log
        description: Records nominal telemetry when Cortex ingester hash rings maintain
          quorum and manage timeseries within safe bounds.
        message: "Cortex ingester ring operating nominally: {{
          outputs.scan_cortex_telemetry.vars.unhealthy_ring_nodes_count }}
          unhealthy nodes, {{
          outputs.scan_cortex_telemetry.vars.in_memory_series_count }} in-memory
          series, {{
          outputs.scan_cortex_telemetry.vars.active_ring_members_total }} active
          members."

  - id: emit_audit_manifest
    type: io.kestra.plugin.core.debug.Return
    description: Emits structured audit manifest for multi-tenant metrics storage
      governance and write SLA records.
    format: |
      {
        "status": "{{ outputs.scan_cortex_telemetry.vars.is_alerting ? 'ALERT' : 'NOMINAL' }}",
        "tenant_id": "{{ inputs.tenant_id }}",
        "unhealthy_ring_nodes": {{ outputs.scan_cortex_telemetry.vars.unhealthy_ring_nodes_count }},
        "in_memory_series": {{ outputs.scan_cortex_telemetry.vars.in_memory_series_count }},
        "active_ring_members": {{ outputs.scan_cortex_telemetry.vars.active_ring_members_total }},
        "flush_errors": {{ outputs.scan_cortex_telemetry.vars.flush_errors_total }},
        "audit_timestamp": "{{ outputs.scan_cortex_telemetry.vars.audit_timestamp }}"
      }

triggers:
  - id: periodic_cortex_audit
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "*/15 * * * *"

Automates continuous health and consensus stability surveillance for Cortex multi-tenant metrics storage ingesters.

How it works

A schedule trigger executes the Cortex ingester audit workflow every 15 minutes.
The Python task queries the Cortex ingester metrics service endpoint on port 8000.
The script extracts hash ring member node states, in-memory active series counts, and chunk flush error totals.
An evaluation gate determines whether unhealthy nodes, series overflow limits, or flush errors breach bounds.
When ingester ring degradation is detected, a high-priority Slack alert is dispatched to the Observability SRE team.
When ingesters maintain healthy ring consensus and stream timeseries to cold storage cleanly, nominal telemetry is logged.
A structured JSON audit manifest is emitted for metrics infrastructure governance and write SLA records.

Architecture diagram

graph TD
    Trigger[Schedule Trigger: 15m] --> ScanMetrics[Python Script: Scrape Cortex Metrics]
    ScanMetrics --> EvalBreach{Unhealthy Nodes > 1 OR Series Overflow?}
    EvalBreach -- Yes --> SlackAlert[Slack: High-Urgency Observability Alert]
    EvalBreach -- No --> LogNominal[Log: Nominal Cortex State]
    SlackAlert --> EmitManifest[Debug Return: Metrics Audit Manifest]
    LogNominal --> EmitManifest

Architecture components

Trigger: Automated schedule polling at 15-minute recurring intervals.
Script Task: Scrapes Cortex ingester Prometheus metrics endpoints using Docker container runners.
Decision Gate: Evaluates whether unhealthy ring member counts or in-memory timeseries volumes exceed limits.
Notification Task: Dispatches structured Slack alerts with tenant identifier and unhealthy node metrics.
Audit Manifest: Emits JSON execution payloads containing Cortex storage health and timestamps.

Key capabilities

Continuous surveillance of Cortex consistent hash ring topologies and member heartbeat health.
Early warning detection for timeseries churn spikes leading to in-memory Series limit exhaustion.
Automated tracking of chunk flush failures to object storage backends like S3 or GCS.
Zero host dependency execution via isolated containerized Python runners.

What you get

Real-time visibility into distributed metrics ingestion quorum, node balance, and memory headroom.
Proactive alerts preventing 500 Internal Server Error ingestion dropouts during write replication quorum loss.
Standardized audit manifests containing exact active member totals and execution timestamps.
Zero-overhead deployment compatible with self-hosted Kubernetes clusters and managed cloud providers.

Who it's for

Observability and monitoring SREs operating multi-tenant Cortex, Mimir, or Thanos backends.
Platform reliability engineers tracking Prometheus remote-write ingestion latency and throughput.
Cloud infrastructure architects designing high-volume timeseries retention systems.
Enterprise DevOps teams validating write availability and timeseries cardinalities across teams.

Why orchestrate this with Kestra

Serverless Docker Isolation: Executes telemetry scraping in ephemeral Docker environments without persistent workers.
Centralized Secrets Governance: Protects sensitive Slack webhooks using encrypted namespace secrets.
Declarative YAML Workflows: Manages metrics control plane surveillance as versioned code in Git repositories.
Comprehensive Audit History: Preserves historical ingester telemetry in searchable Kestra execution logs.

Use cases

Hash Ring Partition Prevention: Alerts engineers before member dropouts drop replication below quorum (RF=3).
Memory Headroom Gating: Monitors active memory series before high cardinality metrics trigger node OOM kills.
Chunk Flush Stall Detection: Identifies slow cloud storage write API rates during hourly chunk compaction cycles.

Inputs

Name	Type	Default	Description
`cortex_metrics_url`	STRING	`http://cortex-ingester.cortex.svc.cluster.local:8000`	Internal HTTP URL exposing Prometheus metrics for Cortex.
`tenant_id`	STRING	`enterprise-telemetry-tenant`	Identifier of the target Cortex tenant partition.
`max_unhealthy_ring_nodes_threshold`	INT	`1`	Maximum allowable unhealthy ring nodes before alerting.
`max_in_memory_series_limit`	INT	`2000000`	Maximum acceptable in-memory series count per ingester.
`alert_on_ring_unhealthy`	BOOL	`true`	Toggle whether ring unhealthy status triggers an alert.

Expected outputs

{{ outputs.scan_cortex_telemetry.vars.unhealthy_ring_nodes_count }}: Count of unhealthy ingester nodes in the ring.
{{ outputs.scan_cortex_telemetry.vars.in_memory_series_count }}: Maximum observed in-memory timeseries volume.
{{ outputs.scan_cortex_telemetry.vars.active_ring_members_total }}: Total count of active healthy ring member nodes.
{{ outputs.scan_cortex_telemetry.vars.is_alerting }}: Boolean flag indicating if metrics storage SLA is breached.
{{ outputs.scan_cortex_telemetry.vars.audit_timestamp }}: ISO-8601 UTC timestamp of the storage audit run.

Prerequisites

Cortex 1.15+ deployed with Prometheus metrics enabled on port 8000.
Network connectivity from the Kestra worker to the Cortex ingester metrics service.
Configured Slack incoming webhook URL with permissions to post in observability alert channels.

Secrets

SLACK_WEBHOOK_URL: Secure webhook URL for routing storage alerts to the Observability SRE Slack channel.

Quick start

Verify that Cortex ingester exposes Prometheus metrics on HTTP port 8000.
Add the SLACK_WEBHOOK_URL secret within your Kestra namespace settings.
Specify your tenant identifier in tenant_id.
Run a manual flow execution via the Kestra UI to validate metric ingestion.
Enable the periodic_cortex_audit schedule trigger to establish automated monitoring.

Common pitfalls and troubleshooting

Gossip Protocol Partitions: Member-list gossip communication issues can report false unhealthy ring node states.
Disk WAL Write Stalls: Slow local NVMe disk writes can back up the ingester write-ahead log (WAL).
Object Storage Rate Limits: Ingester chunk flush errors often stem from cloud S3 rate limiting on PUT requests.
Replication Factor Settings: Ensure ingester ring replication factor (default 3) matches cluster sizing.

How to extend

Automated Ring Token Rebalance: Trigger an administrative ring cleanup API call to forget dead member nodes.
PagerDuty Escalation: Escalate to on-call observability engineers if unhealthy ring nodes persist for 2 cycles.
Tenant Cardinality Breakdown: Fetch per-metric series breakdown from the Cortex ruler API during series overflow alerts.

Cortex Ingester Ring Unhealthy and Series Overflow Sentinel

How it works

Architecture diagram

Architecture components

Key capabilities

What you get

Who it's for

Why orchestrate this with Kestra

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Cortex Ingester Ring Unhealthy and Series Overflow Sentinel

How it works

Architecture diagram

Architecture components

Key capabilities

What you get

Who it's for

Why orchestrate this with Kestra

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

Test rendered Kubernetes manifests with conftest before they merge

Audit the Operating System Inside Your Container Images for End of Life

Multi-Tenant CI/CD, Promote Kestra Flows Across Environments and Tenants Automatically

HashiCorp Consul Raft Leader Stability and Quorum Sentinel

Kubernetes CoreDNS Request Latency Spike and Upstream RTT Sentinel

Lint Commit Messages Against Conventional Commits Before Every Release

More Cloud Blueprints

Kubernetes CNI IPAM Node Pod Subnet Exhaustion Sentinel

Kubernetes CronJob and Batch Job Execution Failure Sentinel

Kubernetes ConfigMap and Secret Hash Drift Sentinel

KMS Key Rotation Compliance Gate

Knative Service Revision Failure and Scale Sentinel

Kubernetes cert-manager ACME Challenge Timeout Sentinel

New to Kestra?