Schedule icon
Get icon
Script icon
If icon
SlackIncomingWebhook icon
Log icon

Kubernetes CSI Volume Attachment Recovery Guard

Diagnose potentially stuck Kubernetes CSI volume attachments by correlating attachment, node, PV, PVC, and pod state.

Categories
Infrastructure

How it works

  1. An optional hourly schedule triggers a read-only inspection; manual execution is recommended first.
  2. Kubernetes Get tasks collect full VolumeAttachment, node, PV, PVC, and pod objects into Kestra internal storage.
  3. A Python task correlates attachment state with node readiness, claim references, pod usage, attachment errors, deletion state, and attachment age.
  4. An If task routes findings to Slack when review is required; otherwise, it logs a clean inspection result.

What you get

  • A structured JSON report of attachments requiring review.
  • Correlation with nodes, persistent volumes, claims, and related pods where resolvable.
  • A configurable threshold for unattached objects.
  • Slack notification when evidence requires investigation.
  • No automatic patches, deletes, or force-detach operations.

Who it's for

  • Kubernetes platform engineers and storage administrators.
  • Site reliability engineers investigating stateful workload outages.
  • Teams operating CSI-backed persistent storage.

Why orchestrate this with Kestra

Kestra combines Kubernetes API calls, data correlation, conditional routing, scheduled execution, and external notifications in a traceable workflow. This avoids a one-off script with no execution history or alert routing.

Prerequisites

  • A Kubernetes API endpoint reachable from the Kestra worker.
  • Read/list permissions for VolumeAttachments, Nodes, PersistentVolumes, PersistentVolumeClaims, and Pods in the selected namespace.
  • Python script task support and access to Kestra internal storage.
  • A Slack incoming webhook if notifications are enabled.

Secrets

  • K8S_MASTER_URL: Kubernetes API server URL.
  • K8S_TOKEN: Kubernetes bearer/OAuth token with the required read-only permissions.
  • SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

  1. Configure the three secrets in the Kestra namespace.
  2. Confirm that the service account or token can list the required Kubernetes resources.
  3. Execute the flow manually with a workload namespace and unattached-age threshold.
  4. Inspect the attachment-review-report.json output and verify findings against the cluster.
  5. Enable the hourly schedule only after a successful test.

How to extend

  • Add provider-specific CSI health checks and storage-driver events.
  • Add an approval step for a documented recovery runbook.
  • Add incident-management integrations alongside Slack.
  • Add post-recovery verification after a separately designed and authorized remediation process.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.