Schedule icon
Webhook icon
Assert icon
Request icon
Get icon
Script icon
Set icon
If icon
Pause icon
Loop icon
Log icon
SlackIncomingWebhook icon

NATS JetStream Abandoned Consumer Retention Reclaim Guard

Detect abandoned NATS JetStream consumers pinning retention, pause for approval, delete stale consumers to reclaim storage, and verify headroom in Kestra.

Categories
DataInfrastructure

In high-throughput NATS JetStream streaming architectures, streams configured with Interest or Limits retention policies enforce that messages cannot be pruned until every registered consumer has acknowledged them. When microservice pods, dynamic test suites, or ephemeral worker processes crash or disconnect without deregistering their consumers, those abandoned consumers leave an unacknowledged sequence floor (ack_floor) pinned at an old stream position.

As upstream publishers continue pushing new messages, the gap between the stream's highest sequence and the abandoned consumer's acknowledgment floor expands. Because JetStream honors the consumer's retention interest, the broker cannot trim expired messages from disk or memory. Eventually, the stream hits its storage ceiling (max_bytes), causing the NATS cluster to reject incoming writes and halt business pipelines.

This blueprint provides an automated retention reclaim guard. It queries JetStream management APIs for consumer subscription status and acknowledgment floors, tracks consecutive inactive observations across runs in Kestra KV to avoid false positives, gates destructive deletions behind a human approval Pause, executes targeted $JS.API.CONSUMER.DELETE commands, verifies sequence advancement and storage reclamation, and dispatches detailed Slack audit reports.

How it works

  1. Pre-flight Assertion: Validates operational inputs (io.kestra.plugin.core.execution.Assert), verifying valid stream names, positive inactivity intervals, and non-negative lag thresholds.
  2. Stream Telemetry Collection: get_stream_info (io.kestra.plugin.nats.core.Request) queries $JS.API.STREAM.INFO.<stream> to retrieve retention mode, current byte usage, and first/last sequence numbers.
  3. Consumer Topology Discovery: list_consumers (io.kestra.plugin.nats.core.Request) queries $JS.API.CONSUMER.LIST.<stream> to inspect all durable and ephemeral consumers, pulling their acknowledgment floors (ack_floor), active subscriptions (num_waiting), push bindings (push_bound), and pending message counts.
  4. Historical State Retrieval: get_previous_audit_state (io.kestra.plugin.core.kv.Get) retrieves the previous audit manifest from Kestra KV store under nats_retention_guard_{stream_name}.
  5. Triage and Abandonment Classification: A Python inspection script (inspect_abandoned_consumers) cross-references consumer activity against historical records. A consumer is classified as ABANDONED_RETENTION_PINNED only when it has zero active subscriptions, an inactive duration exceeding inactivity_threshold_seconds, stationary acknowledgment sequence across min_consecutive_inactive_runs, and sequence lag exceeding min_retention_lag_messages.
  6. Cross-Run State Tracking: save_audit_state (io.kestra.plugin.core.kv.Set) updates the Kestra KV store with consecutive inactive run counters for each consumer, enabling trend-based detection.
  7. Human Approval Gate: When abandoned consumers are detected and dry_run is disabled, the workflow halts at pause_for_reclaim_approval (io.kestra.plugin.core.flow.Pause), presenting operators with candidate consumer names, pinned sequence floors, and estimated storage footprints.
  8. Write-Path Consumer Deletion: Upon operator approval, delete_abandoned_consumers (io.kestra.plugin.core.flow.Loop) iterates through each abandoned consumer and sends $JS.API.CONSUMER.DELETE.<stream>.<consumer> requests to the NATS broker.
  9. Post-Remediation Verification: The flow re-queries $JS.API.CONSUMER.LIST.<stream> and $JS.API.STREAM.INFO.<stream> to prove consumer deregistration and confirm that stream retention sequence floors advanced.
  10. Incident and Audit Alerting: Structured Slack alerts are dispatched (notify_slack_audit) with recovery metrics, while execution errors trigger dedicated triage notifications.

What you get

  • Automated discovery of abandoned ephemeral and durable consumers pinning stream retention.
  • Protection against broker disk and memory saturation caused by dead consumer sequence floors.
  • Differential trend analysis via Kestra KV preventing premature deletion of restarting workers.
  • Operator approval gates and dry-run simulation to review deletion candidates safely.
  • Post-remediation verification confirming that consumers were deleted and storage headroom recovered.
  • Structured Slack notifications detailing reclaimed bytes and updated stream sequence bounds.

Who it's for

  • Platform Engineers and SREs maintaining production NATS JetStream messaging clusters.
  • Streaming Data Engineers operating microservice event buses and pub/sub pipelines.
  • DevOps Teams automating infrastructure hygiene, resource cleanup, and day-2 operations.

Why orchestrate this with Kestra

NATS CLI commands like nats consumer rm require manual terminal execution and cannot track consecutive inactive observations across time, enforce approval gates, or verify storage recovery automatically. Custom shell scripts and cron jobs lack cross-run state persistence, secret masking, and centralized audit trails.

Orchestrating this guard with Kestra provides native NATS request/reply integration, built-in KV store state tracking, human-in-the-loop pause gates, comprehensive execution logging, and automated multi-channel alerting.

Prerequisites

  • A reachable NATS server (version 2.9 or higher) with JetStream enabled.
  • Target stream created with Interest or Limits retention policy.
  • NATS user credentials (if authentication is enabled) with permissions to query $JS.API.> subjects.
  • A Slack Incoming Webhook URL to receive audit notifications and failure alerts.

Secrets

  • SLACK_WEBHOOK_URL: Slack Incoming Webhook URL used to dispatch audit summaries and emergency incident alerts.
  • NATS_RETENTION_GUARD_WEBHOOK_KEY: Secret authentication key protecting the on-demand webhook trigger.

Quick start

  1. Configure SLACK_WEBHOOK_URL and NATS_RETENTION_GUARD_WEBHOOK_KEY in your Kestra namespace secrets.
  2. Import this blueprint and set nats_url and stream_name to match your target NATS environment.
  3. Run a manual execution with dry_run: true to record an initial baseline and review identified consumer candidates.
  4. Enable the periodic_retention_guard schedule trigger to run automated retention audits every 2 hours.
  5. When ready for automated remediation, set dry_run: false to allow operator-approved consumer deletions.

How to extend

  • Kubernetes Pod Lifecycle Correlation: Add an io.kestra.plugin.kubernetes.pod.List task to check whether consumer names correspond to terminated Kubernetes pods before deletion.
  • PagerDuty High-Priority Incident Escalation: Integrate io.kestra.plugin.pagerduty.notifications.PagerDutyAlert to page on-call engineers when stream storage usage exceeds 90%.
  • Dead-Letter Replay Queue: Publish consumer acknowledgment floor metadata to a dedicated quarantine stream prior to deletion for compliance auditing.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.