Grafana Mimir Ingester Sentinel

id: mimir-ingester-compaction-and-write-error-sentinel
namespace: company.infrastructure

inputs:
  - id: mimir_ingester_endpoint
    type: STRING
    displayName: Grafana Mimir Ingester Metrics Endpoint
    description: Base URL of the Grafana Mimir ingester Prometheus metrics endpoint.
    defaults: "http://localhost:8080"

  - id: max_allowed_compaction_failures
    type: INT
    displayName: Maximum Allowed Compaction Failures
    description: Tolerance threshold for TSDB head compaction failures before
      raising a critical incident alert.
    defaults: 0

  - id: send_slack_alert
    type: BOOL
    displayName: Dispatch Slack Incident Alert
    description: Whether to transmit real-time Slack notifications upon TSDB
      compaction failure or WAL corruption.
    defaults: true

triggers:
  - id: mimir_audit_schedule
    type: io.kestra.plugin.core.trigger.Schedule
    description: Recurring audit polling Grafana Mimir ingester health every 15 minutes.
    cron: "*/15 * * * *"
    disabled: true

tasks:
  - id: probe_ingester_metrics
    type: io.kestra.plugin.core.http.Request
    description: Query Grafana Mimir ingester Prometheus metrics endpoint for
      compaction and WAL statistics.
    uri: "{{ inputs.mimir_ingester_endpoint }}/metrics"
    method: GET
    headers:
      Accept: "text/plain"
    options:
      allowFailed: true
      timeout:
        connectTimeout: PT10S
        readIdleTimeout: PT30S
    retry:
      type: constant
      interval: PT5S
      maxAttempts: 3
    allowFailure: true

  - id: evaluate_compaction_health
    type: io.kestra.plugin.scripts.python.Script
    description: Interrogate Prometheus metrics text, extract compaction failures
      and WAL corruptions, and classify health state.
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    inputFiles:
      metrics.txt: "{{ outputs.probe_ingester_metrics.body }}"
    script: |
      import re
      import json

      max_allowed = int({{ inputs.max_allowed_compaction_failures }})

      try:
          with open("metrics.txt", "r") as f:
              raw = f.read()
      except Exception:
          raw = ""

      def extract_metric(pattern):
          matches = re.findall(pattern, raw)
          total = 0.0
          for m in matches:
              try:
                  total += float(m)
              except ValueError:
                  pass
          return total

      compaction_failures = extract_metric(r'mimir_ingester_tsdb_head_compaction_failed_total(?:\{[^}]*\})?\s+([0-9\.]+)')
      wal_corruptions = extract_metric(r'mimir_ingester_wal_corruptions_total(?:\{[^}]*\})?\s+([0-9\.]+)')
      out_of_order = extract_metric(r'mimir_ingester_tsdb_samples_out_of_order_total(?:\{[^}]*\})?\s+([0-9\.]+)')
      memory_series = extract_metric(r'mimir_ingester_memory_series(?:\{[^}]*\})?\s+([0-9\.]+)')

      if compaction_failures == 0 and wal_corruptions == 0:
          if out_of_order > 100:
              classification = "OUT_OF_ORDER_SAMPLE_WARNING"
          else:
              classification = "INGESTION_OPTIMAL"
      elif compaction_failures <= max_allowed and wal_corruptions == 0:
          classification = "OUT_OF_ORDER_SAMPLE_WARNING"
      else:
          classification = "CRITICAL_COMPACTION_FAILURE"

      output_payload = {
          "outputs": {
              "compaction_failures": int(compaction_failures),
              "wal_corruptions": int(wal_corruptions),
              "out_of_order_samples": int(out_of_order),
              "active_memory_series": int(memory_series),
              "classification": classification
          }
      }
      print(f"::{json.dumps(output_payload)}::")

  - id: classify_ingester_status
    type: io.kestra.plugin.core.log.Log
    description: Log evaluated Grafana Mimir ingester health metrics and status
      classification.
    message: |
      Mimir Endpoint: {{ inputs.mimir_ingester_endpoint }}
      Compaction Failures: {{ outputs.evaluate_compaction_health.vars.compaction_failures }}
      WAL Corruptions: {{ outputs.evaluate_compaction_health.vars.wal_corruptions }}
      Out-of-Order Samples: {{ outputs.evaluate_compaction_health.vars.out_of_order_samples }}
      Active Memory Series: {{ outputs.evaluate_compaction_health.vars.active_memory_series }}
      Classification: {{ outputs.evaluate_compaction_health.vars.classification }}

  - id: route_ingester_gate
    type: io.kestra.plugin.core.flow.Switch
    description: Route flow execution based on evaluated Grafana Mimir ingester
      health classification.
    value: "{{ outputs.evaluate_compaction_health.vars.classification }}"
    cases:
      INGESTION_OPTIMAL:
        - id: log_optimal_state
          type: io.kestra.plugin.core.log.Log
          description: Log nominal verification confirming TSDB head blocks are compacting
            cleanly.
          message: "OPTIMAL: Grafana Mimir ingester TSDB compaction and WAL writes are
            operating without errors."

      OUT_OF_ORDER_SAMPLE_WARNING:
        - id: log_sample_advisory
          type: io.kestra.plugin.core.log.Log
          description: Log advisory warning for out-of-order sample rejections or minor
            compaction issues.
          message: "WARNING: Elevated out-of-order sample rejections ({{
            outputs.evaluate_compaction_health.vars.out_of_order_samples }})
            detected on Mimir ingester."

      CRITICAL_COMPACTION_FAILURE:
        - id: alert_slack_compaction_failure
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          description: Dispatch urgent incident alert to observability engineering channel.
          url: "{{ secret('SLACK_WEBHOOK_URL') }}"
          payload: |
            {
              "text": ":rotating_light: *CRITICAL Grafana Mimir Ingester Compaction Failure Alert*\n*Endpoint:* `{{ inputs.mimir_ingester_endpoint }}`\n*Compaction Failures:* `{{ outputs.evaluate_compaction_health.vars.compaction_failures }}`\n*WAL Corruptions:* `{{ outputs.evaluate_compaction_health.vars.wal_corruptions }}`\n*Out-of-Order Samples:* `{{ outputs.evaluate_compaction_health.vars.out_of_order_samples }}`\n*Active Series:* `{{ outputs.evaluate_compaction_health.vars.active_memory_series }}`\n*Execution:* <{{ flow.url }}|{{ execution.id }}>"
            }

        - id: remediation_fail_gate
          type: io.kestra.plugin.core.execution.Fail
          description: Fail execution to trigger incident escalation and prevent permanent
            metric sample drops.

  - id: final_audit_log
    type: io.kestra.plugin.core.log.Log
    description: Record immutable audit log entry documenting Grafana Mimir ingester
      health verification.
    message: |
      [AUDIT] Grafana Mimir Ingester Compaction and Write Error Sentinel completed.
      Endpoint: {{ inputs.mimir_ingester_endpoint }}
      Compaction Failures: {{ outputs.evaluate_compaction_health.vars.compaction_failures }}
      WAL Corruptions: {{ outputs.evaluate_compaction_health.vars.wal_corruptions }}
      Verdict: {{ outputs.evaluate_compaction_health.vars.classification }}

Automated metrics infrastructure sentinel that interrogates Grafana Mimir ingester Prometheus metrics on a recurring schedule to audit TSDB head block compaction health, detect WAL corruptions, classify metric loss severity, and dispatch Slack incident alerts before out-of-order sample rejections create blind spots in monitoring dashboards.

How it works

A schedule trigger activates every fifteen minutes to audit the Grafana Mimir ingester component.
An HTTP request task queries the ingester's /metrics Prometheus endpoint to gather real-time compaction and write error counters.
An embedded Python script parses metric lines, extracting mimir_ingester_tsdb_head_compaction_failed_total, mimir_ingester_wal_corruptions_total, out-of-order samples, and active memory series counts.
An informational log task records evaluated ingester metrics and current storage classification.
A Switch flow control task branches execution according to the evaluated status:
- INGESTION_OPTIMAL: Confirms zero compaction errors and logs nominal operational telemetry.
- OUT_OF_ORDER_SAMPLE_WARNING: Emits a non-blocking advisory notification when client sample clock skews cause rejected metrics.
- CRITICAL_COMPACTION_FAILURE: Dispatches a detailed Slack incident webhook with failure counts, WAL corruption tallies, execution deep links, and halts execution via a failure gate.
An immutable final audit log records the comprehensive health verdict and error metrics in Kestra execution history.

What you get

Real-time TSDB health monitoring catching compaction stalls before ingester pods crash or run out of memory.
Automated detection of write-ahead log (WAL) corruptions and disk persistence failures.
Immediate Slack notifications alerting SRE and platform engineering teams to metric ingestion degradation.
Automated execution failure gates that halt dependent monitoring pipelines when Prometheus metric stores degrade.
Complete, immutable audit records tracking TSDB head block compaction stability over time.

Who it's for

Site Reliability Engineers (SREs) and Platform Engineers maintaining large-scale Grafana Mimir or Cortex clusters.
Observability and DevOps Teams managing long-term Prometheus metrics storage tiers.
Monitoring Architects ensuring continuous SLA and SLO metric fidelity across Kubernetes environments.

Why orchestrate this with Kestra

Declarative YAML orchestrates metrics harvesting, text parsing, and incident dispatching without dedicated sidecar daemons.
Native secret masking ensures Slack webhook URLs and administrative credentials remain confidential during automated runs.
Built-in conditional branching enables granular escalation policies tailored to metric loss thresholds.
Centralized execution timelines provide instant diagnostic visibility into historical TSDB failure patterns.

Prerequisites

Running Grafana Mimir cluster or standalone container with the ingester component exposing Prometheus metrics (default port 8080).
Network reachability between Kestra and the Mimir ingester endpoint.

Secrets

SLACK_WEBHOOK_URL: Slack Incoming Webhook URL used to transmit critical compaction failure incident alerts.

Quick start

Configure the SLACK_WEBHOOK_URL secret in your Kestra namespace or environment variables.
Import this blueprint into your Kestra instance under company.infrastructure.
Run a manual test execution pointing to your Mimir ingester endpoint to verify metric connectivity and parsing.
Enable the mimir_audit_schedule trigger to maintain continuous 15-minute ingester monitoring.

How to extend

Add an automated pod restart or rolling compaction trigger in the remediation branch using Kestra's Kubernetes plugin.
Correlate compaction failures with ingester persistent volume disk usage and I/O latency metrics.
Route separate alerts to PagerDuty or Opsgenie when WAL corruptions threaten metric durability.

Grafana Mimir Ingester Compaction and Write Error Sentinel

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Grafana Mimir Ingester Compaction and Write Error Sentinel

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

Enforce default SSE-S3 encryption on a MinIO bucket

Enforce a MinIO bucket lifecycle expiry and noncurrent version policy

Scheduled MinIO Bucket Retention Cleanup

Memcached Slab Eviction Sentinel

Prune noncurrent object versions billed forever behind a versioned bucket

Catch MinIO Replication Drift Before Objects Go Missing on the Target

New to Kestra?