Clever Cloud Application State Drift Sentinel
Fetch a Clever Cloud application's state, alert Slack when it drifts from the expected state you declare, and log a governance audit trail.
Audits Kubernetes PersistentVolumeClaim disk usage via Prometheus, identifies impending volume exhaustion, and alerts SREs on Slack with remediation commands.
Audits Kubernetes PersistentVolumeClaim (PVC) storage utilization telemetry via Prometheus metrics, detecting volumes rapidly approaching disk capacity limits and alerting SRE teams on Slack with instant kubectl volume expansion commands before pod evictions occur.
kubelet_volume_stats_used_bytes and kubelet_volume_stats_capacity_bytes) across the target Kubernetes namespace.kubectl patch pvc expansion snippets and dispatches structured alerts directly to designated Slack channels.flowchart TD
A[periodic_pvc_storage_audit Schedule Trigger] --> B[scan_pvc_storage_metrics Python Script]
B --> C[check_pvc_exhaustion_threshold Flowable If]
C -- Capacity Breached --> D[notify_slack_storage_sre Webhook]
C -- Storage Nominal --> E[log_nominal_storage_state Log]
D --> E
E --> F[export_storage_manifest Return]
periodic_pvc_storage_audit): Drives scheduled recurring surveillance cycles every 15 minutes without human intervention.scan_pvc_storage_metrics): Queries Prometheus APIs, computes volume utilization ratios, and structures diagnostic indicators via Kestra outputs.check_pvc_exhaustion_threshold): Evaluates operational safety boundaries using deterministic boolean branching.notify_slack_storage_sre): Dispatches rich, actionable incident messages with pre-computed kubectl patch commands directly to team channels.export_storage_manifest): Emits structured capacity payloads for SRE dashboards or long-term capacity planning.kubectl patch pvc commands with calculated 1.5x expansion sizing to accelerate incident resolution.kubectl patch expansion commands delivered directly to Slack.kubectl patch execution steps or Jira ticket creation within the same declarative YAML flow.| Name | Type | Default | Description |
|---|---|---|---|
prometheus_endpoint |
STRING |
http://prometheus-k8s.monitoring:9090 |
Base HTTP URL of the Prometheus server scraping kubelet volume statistics. |
cluster_identifier |
STRING |
k8s-production-us-east-1 |
Cluster identifier label utilized within Prometheus metrics queries. |
target_namespace |
STRING |
production |
Target Kubernetes namespace under audit, or 'all' to inspect all persistent volumes. |
warning_threshold_pct |
FLOAT |
80.0 |
Disk usage percentage threshold triggering a warning notification. |
critical_threshold_pct |
FLOAT |
90.0 |
Disk usage percentage threshold triggering an emergency priority incident. |
lookback_burn_hours |
INT |
6 |
Hours of historical disk consumption used to calculate burn rates. |
alert_on_exhaustion |
BOOL |
True |
Toggle whether Slack webhook alerts are dispatched upon detecting elevated capacity usage. |
{{ outputs.scan_pvc_storage_metrics.vars.total_pvcs_scanned }}: Number of persistent volume claims scanned during the audit run.{{ outputs.scan_pvc_storage_metrics.vars.warning_pvc_count }}: Number of PVCs currently between warning and critical capacity limits.{{ outputs.scan_pvc_storage_metrics.vars.critical_pvc_count }}: Number of PVCs currently exceeding the critical capacity ceiling.{{ outputs.scan_pvc_storage_metrics.vars.max_utilization_pct }}: Peak utilization percentage recorded across all inspected volumes.{{ outputs.scan_pvc_storage_metrics.vars.top_exhaustion_pvc }}: Identifier (namespace/pvc) of the highest-utilization volume.{{ outputs.scan_pvc_storage_metrics.vars.remediation_command }}: Copy-pasteable kubectl patch command to immediately expand the storage size.{{ outputs.scan_pvc_storage_metrics.vars.is_alerting }}: Boolean flag indicating whether any volume exceeded configured safety thresholds.allowVolumeExpansion: true).kubelet_volume_stats_* metrics endpoints.SLACK_WEBHOOK_URL: Webhook URL for posting storage incident alerts into Slack.SLACK_WEBHOOK_URL.company.platform namespace.prometheus_endpoint to match your cluster's Prometheus internal DNS address.StorageClass has allowVolumeExpansion: true, otherwise kubectl patch will fail to resize the volume.container_fs_usage_bytes.kubectl apply task via io.kestra.plugin.kubernetes.cli.Kubectl to expand storage automatically when critical.io.kestra.plugin.pagerduty.Incident step to wake on-call engineers when PVC utilization exceeds 95%.