Schedule icon
Webhook icon
Assert icon
Get icon
Script icon
Set icon
If icon
SlackIncomingWebhook icon
Pause icon
Publish icon
Log icon

Redis Stale Distributed Lock and Deadlock Lease Reclamation Guard

Audit Redis distributed locks, detect infinite TTLs and deadlocks, pause for approval, delete stale locks, verify reclamation, and alert on Slack.

Categories
DataInfrastructure

Distributed locking patterns in Redis (including Redlock algorithms, Celery worker locks, BullMQ mutexes, and Spring ShedLock) coordinate critical mutual exclusion across distributed microservices. However, worker process crashes, out-of-memory container terminations, and unhandled runtime exceptions frequently orphan lock keys. When a client sets a lock key without an expiration (TTL = -1) or crashes before releasing a long-duration lease, subsequent pipeline jobs stall indefinitely, creating severe distributed task gridlock.

This blueprint implements an automated Redis distributed lock sentinel and reclamation guard. It tracks active lock keys across execution runs via Kestra Key-Value store, detects infinite TTLs and abandoned leases, isolates deadlocks into an audit quarantine certificate, pauses for operator approval, safely deletes stuck lock keys, broadcasts release events to Redis Pub/Sub, verifies absence in the cluster, and alerts Slack.

How it works

  1. The scheduled_lock_guard schedule trigger executes the audit every 15 minutes, while on_demand_deadlock_webhook provides an on-demand webhook entry point for external worker monitoring alarms or on-call runbooks.
  2. The validate_inputs task (io.kestra.plugin.core.execution.Assert) verifies that connection properties and threshold constraints are positive and well-formed.
  3. The fetch_historical_locks task (io.kestra.plugin.core.kv.Get) retrieves the previous scan snapshot from Kestra KV store under key redis_active_locks_state to track multi-run lease continuity.
  4. The triage_distributed_locks task (io.kestra.plugin.scripts.python.Script) connects to Redis, scans keys matching lock_key_pattern, extracts holder tokens, inspects remaining TTLs, measures idle times, and compares against historical snapshots:
    • INFINITE_TTL_DEADLOCK: Flagged when a lock has no expiration (TTL == -1), posing immediate risk of permanent worker starvation.
    • STALE_LEASE_EXCEEDED: Flagged when a lock with identical holder token has persisted across consecutive runs exceeding stale_lock_threshold_seconds.
    • HEALTHY_ACTIVE: In-flight lock within operational lease parameters.
  5. The persist_active_locks_state task (io.kestra.plugin.core.kv.Set) updates Kestra KV store with current active lock telemetry for trend tracking on subsequent runs.
  6. The branch_on_deadlocks task (io.kestra.plugin.core.flow.If) evaluates whether deadlocked locks were identified:
    • Archives diagnosed deadlock records to redis_deadlock_quarantine_manifest in Kestra KV.
    • Dispatches an incident alert to Slack detailing key names, holder tokens, and detected hold durations.
    • When auto_remediate is enabled and dry_run is false, triggers the interactive pause_for_reclamation_approval gate (io.kestra.plugin.core.flow.Pause) requiring operator confirmation.
    • Upon approval, delete_deadlocked_keys (io.kestra.plugin.scripts.python.Script) releases each deadlocked key only if its current value still matches the holder token observed at detection time (a Redlock-safe compare-and-delete via Lua EVAL), skipping any key that a new legitimate worker re-acquired while the approval gate was open.
    • If publish_release_event is enabled, broadcast_lock_release (io.kestra.plugin.redis.pubsub.Publish) publishes structured JSON events to release_channel to notify waiting worker subscribers.
    • The verify_locks_reclaimed task (io.kestra.plugin.scripts.python.Script) re-queries Redis to assert every target key is absent, raising an error if any lock lingers.
    • Posts a confirmation alert to Slack upon successful verification.
  7. An errors handler catches unexpected execution errors and pages on-call engineers via Slack.

What you get

  • Automated discovery and categorization of Redis distributed locks trapped in infinite TTL or stale lease states.
  • Historical multi-run trend detection distinguishing normal long-running tasks from abandoned crashed worker locks.
  • Interactive human-in-the-loop approval gate before executing any destructive key deletions.
  • Native Redis string deletion combined with Pub/Sub release broadcasts to immediately awaken blocked workers.
  • Post-remediation verification asserting that released lock keys have ceased to exist in Redis.
  • Quarantined audit trail preserved in Kestra KV for compliance and post-mortem incident analysis.
  • Context-rich Slack notifications with token identifiers, hold times, and cluster endpoints.

Who it's for

  • Platform Engineers and Site Reliability Engineers managing Redis infrastructure and distributed worker fleets.
  • Backend and Distributed Systems Engineers maintaining Celery, BullMQ, Sidekiq, or Spring ShedLock task pipelines.
  • Data Engineers orchestrating distributed extract-transform-load (ETL) jobs requiring mutex coordination.

Why orchestrate this with Kestra

Distributed locks are inherently stateful and hazardous to clear blindly. A naive cron job running redis-cli del risks evicting active jobs that are simply taking longer than average. Kestra solves this by persisting lock observation history across runs in atomic namespace KV storage, gating destructive operations behind interactive Pause approval gates, broadcasting recovery events to Pub/Sub, and asserting post-action cleanliness with verified verification queries.

Prerequisites

  • Reachable Redis instance (version 6.x, 7.x, or compatible KeyDB/Valkey).
  • Network connectivity from Kestra workers to the Redis port.
  • Slack Incoming Webhook URL configured for incident triage notifications.

Secrets

  • REDIS_PASSWORD: Password for authenticating to the target Redis instance (leave empty if unauthenticated).
  • REDIS_LOCK_GUARD_WEBHOOK_KEY: Secret authentication key for the on-demand webhook trigger.
  • SLACK_WEBHOOK_URL: Slack Incoming Webhook URL for operational notifications and alerts.

Quick start

  1. Configure REDIS_PASSWORD, REDIS_LOCK_GUARD_WEBHOOK_KEY, and SLACK_WEBHOOK_URL in your Kestra namespace secrets.
  2. Run an initial manual execution with default parameters (dry_run: true) to audit current lock keys and inspect detected TTLs without deleting keys.
  3. Review the execution logs, Slack triage notification, and Kestra KV store entries under redis_active_locks_state.
  4. Run with auto_remediate: true and dry_run: false. When deadlocks are identified, approve the pause_for_reclamation_approval gate in the Kestra UI to release the stuck locks.
  5. Enable the scheduled_lock_guard schedule trigger to maintain continuous automated deadlock protection.

How to extend

  • Customize lock_key_pattern to isolate specific service prefixes (e.g., celery-task-meta:* or shedlock:*).
  • Add a downstream webhook task to trigger an automated pod restart for the deadlocked worker service.
  • Connect PagerDuty or Opsgenie notifications for automated escalation when deadlocked_count exceeds a critical threshold.
  • Route quarantined lock manifests to Amazon S3 or Google Cloud Storage for permanent regulatory audit retention.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.