Harbor Registry Garbage Collection Execution Stall Sentinel
Audits Harbor enterprise container registry telemetry to detect stalled garbage collection jobs, lock freezes, and storage bloat.
Monitor Kubernetes DaemonSet scheduling and node coverage with Kestra. Query AppsV1 API, open GitHub incident issues, and alert SRE on Slack.
Kubernetes DaemonSets ensure that an essential system agent, such as a CNI networking plugin, log collector (FluentBit, Datadog), or security monitor (Falco), runs on every worker node across the cluster. When worker nodes run out of schedulable CPU/memory, have untolerated taints, or fail image pull secrets, the DaemonSet enters a scheduling deficit where desiredNumberScheduled exceeds numberReady. Nodes without running security or logging daemons create blind spots and network partition risks. This blueprint inspects Kubernetes DaemonSets cluster-wide, flags unready node deficits, opens a prioritized GitHub SRE incident ticket, and alerts operations engineers on Slack.
periodic_daemonset_check (io.kestra.plugin.core.trigger.Schedule) initiates the audit every 15 minutes (shipped disabled for safe verification).probe_daemonset_status (io.kestra.plugin.scripts.python.Script) uses the official Kubernetes Python SDK to read DaemonSet status objects across all namespaces.branch_on_deficit (io.kestra.plugin.core.flow.If) evaluates whether any DaemonSets have desired > ready or unavailable > 0:then: Automatically files a GitHub tracking issue via create_daemonset_incident_issue (io.kestra.plugin.github.issues.Create) and alerts the Slack channel via alert_sre_slack (io.kestra.plugin.slack.notifications.SlackIncomingWebhook).else: Emits an informational log entry (log_fleet_healthy) confirming full node coverage.log_audit_complete (io.kestra.plugin.core.log.Log) records the audit outcome.errors block alerts Slack if cluster authentication or network connectivity fails.Kestra coordinates cluster API communication, multi-daemon health comparison, conditional escalation, and chat alerting in a single declarative workflow. It manages kubeconfig credentials securely, provides complete execution history, and eliminates the need for separate standalone monitoring daemons.
daemonsets.KUBECONFIG (or in-cluster ServiceAccount): Authentication credentials for Kubernetes API.GITHUB_TOKEN: GitHub token used to open incident issues.SLACK_WEBHOOK: Incoming Webhook URL for cluster alerts.namespace (STRING, default ""): Optional namespace filter (leave empty to scan cluster-wide).target_repository (STRING, default my-org/kubernetes-cluster-operations): Target GitHub repository for logging incident issues.outputs.probe_daemonset_status.vars.total_inspected: Total number of DaemonSets audited.outputs.probe_daemonset_status.vars.deficit_count: Count of DaemonSets experiencing scheduling deficits.outputs.probe_daemonset_status.vars.degraded_daemonsets: List of degraded DaemonSet objects (namespace, name, desired, ready, unavailable).GITHUB_TOKEN and SLACK_WEBHOOK in your Kestra namespace.target_repository to match your operations tracker.periodic_daemonset_check trigger for continuous scheduling health monitoring.k8s-app=kube-dns) to prioritize networking components.