Get icon
Script icon
Process icon
Set icon
If icon
SlackIncomingWebhook icon
PagerDutyAlert icon
Log icon
Schedule icon
Webhook icon

Redis memory fragmentation ratio and key eviction circuit breaker

Guard Redis against fragmentation and cache thrashing. Remediate with MEMORY PURGE or active defragmentation, track state in KV, and notify on-call.

Categories
DataInfrastructure

Prevent Redis instances from experiencing out-of-memory (OOM) kills or cache thrashing due to allocator fragmentation and runaway eviction spikes.

How it works

  1. Baseline Extraction: Reads the prior audit record from Kestra's internal KV store using io.kestra.plugin.core.kv.Get to establish baseline eviction counters and timestamps.
  2. RESP Diagnostic Probe: Connects directly to Redis over the native TCP socket, issuing INFO memory, INFO stats, and CONFIG GET activedefrag. Parses used_memory, used_memory_rss, mem_fragmentation_ratio, and evicted_keys.
  3. Circuit Breaker Evaluation:
    • Calculates the delta between resident set size (RSS) and allocator memory.
    • Compares the fragmentation ratio against warning (default 1.5) and critical (default 1.8) thresholds, gated by minimum fragmented megabytes (default 64 MiB) to avoid false alarms on idle instances.
    • Derives eviction velocity (keys evicted per second) against previous runs to detect cache thrashing caused by improper TTLs or undersized maxmemory.
  4. Automated Remediation: When critical fragmentation is detected and auto_remediate is enabled:
    • Runs MEMORY PURGE to reclaim unused jemalloc dirty pages back to the operating system.
    • If the ratio remains critical or the configured strategy is ACTIVE_DEFRAG, enables background defragmentation (CONFIG SET activedefrag yes).
    • Measures post-remediation memory and ratio to verify allocator recovery.
  5. Audit State Persistence: Writes the comprehensive audit record into Kestra's KV store using io.kestra.plugin.core.kv.Set.
  6. Multi-Channel Alerting: Branches via io.kestra.plugin.core.flow.If to dispatch detailed alert blocks to Slack and trigger incident escalation in PagerDuty via io.kestra.plugin.pagerduty.PagerDutyAlert.

What you get

  • Real-time detection of runaway memory fragmentation and cache thrashing.
  • Automated self-healing via MEMORY PURGE and CONFIG SET activedefrag yes.
  • Continuous state tracking in Kestra's KV store without external metric databases.
  • Tiered alert routing with Slack notifications and PagerDuty incident triggers.

Who it's for

  • Site Reliability Engineers (SREs) and Database Administrators managing Redis clusters.
  • Platform teams seeking automated circuit breaking for in-memory caching infrastructure.

Why orchestrate this with Kestra

Redis memory management often requires coordinating multiple operational steps: reading state, computing velocity across time windows, executing multi-step remediations, recording audit trails, and paging on-call engineers. Orchestrating this inside Kestra provides built-in statefulness via the KV store, declarative credential security, full execution auditability, and native multi-channel alerting.

Prerequisites

  • Kestra 2.0 or later with Python script and notification plugins.
  • Network connectivity from the Kestra worker to the Redis instance.
  • Redis permissions to execute INFO, CONFIG GET, CONFIG SET, and MEMORY PURGE.

Secrets

  • SLACK_WEBHOOK_URL: Incoming webhook URL for operational notifications.
  • PAGERDUTY_ROUTING_KEY: Service integration key for PagerDuty Events API v2.
  • REDIS_CIRCUIT_BREAKER_WEBHOOK_KEY: Secret authentication key for the webhook trigger.

Quick start

  1. Configure SLACK_WEBHOOK_URL in Kestra Secrets.
  2. Set redis_host and redis_port to match your Redis deployment.
  3. Run an initial manual execution to verify connectivity and seed the initial KV baseline.
  4. Enable the periodic_audit_schedule trigger for continuous 5-minute monitoring.

How to extend

  • Add AWS ElastiCache or Redis Enterprise REST API endpoints to scale across clusters.
  • Integrate CONFIG SET maxmemory adjustments for dynamic cache scaling.
  • Route metrics to Prometheus or Datadog using Kestra's HTTP request plugin.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.