Schedule icon
Script icon
If icon
Create icon
SlackIncomingWebhook icon
Log icon

Kubernetes DaemonSet Node Scheduling Deficit Sentinel

Monitor Kubernetes DaemonSet scheduling and node coverage with Kestra. Query AppsV1 API, open GitHub incident issues, and alert SRE on Slack.

Categories
CloudInfrastructure

Kubernetes DaemonSets ensure that an essential system agent, such as a CNI networking plugin, log collector (FluentBit, Datadog), or security monitor (Falco), runs on every worker node across the cluster. When worker nodes run out of schedulable CPU/memory, have untolerated taints, or fail image pull secrets, the DaemonSet enters a scheduling deficit where desiredNumberScheduled exceeds numberReady. Nodes without running security or logging daemons create blind spots and network partition risks. This blueprint inspects Kubernetes DaemonSets cluster-wide, flags unready node deficits, opens a prioritized GitHub SRE incident ticket, and alerts operations engineers on Slack.

How it works

  1. periodic_daemonset_check (io.kestra.plugin.core.trigger.Schedule) initiates the audit every 15 minutes (shipped disabled for safe verification).
  2. probe_daemonset_status (io.kestra.plugin.scripts.python.Script) uses the official Kubernetes Python SDK to read DaemonSet status objects across all namespaces.
  3. branch_on_deficit (io.kestra.plugin.core.flow.If) evaluates whether any DaemonSets have desired > ready or unavailable > 0:
    • then: Automatically files a GitHub tracking issue via create_daemonset_incident_issue (io.kestra.plugin.github.issues.Create) and alerts the Slack channel via alert_sre_slack (io.kestra.plugin.slack.notifications.SlackIncomingWebhook).
    • else: Emits an informational log entry (log_fleet_healthy) confirming full node coverage.
  4. log_audit_complete (io.kestra.plugin.core.log.Log) records the audit outcome.
  5. The errors block alerts Slack if cluster authentication or network connectivity fails.

What you get

  • Automated, scheduled auditing of Kubernetes infrastructure daemons.
  • Zero silent node coverage gaps for critical security and monitoring agents.
  • Consolidated GitHub incident tickets with namespace names, desired counts, and unavailable numbers.
  • Real-time Slack notifications with direct ticket hyperlinks.

Who it's for

  • Platform Engineers and Kubernetes Administrators managing production clusters.
  • Site Reliability Engineers ensuring consistent agent coverage across worker fleets.
  • Security Engineers verifying runtime protection agent distribution.

Why orchestrate this with Kestra

Kestra coordinates cluster API communication, multi-daemon health comparison, conditional escalation, and chat alerting in a single declarative workflow. It manages kubeconfig credentials securely, provides complete execution history, and eliminates the need for separate standalone monitoring daemons.

Prerequisites

  • A Kubernetes cluster (EKS, GKE, AKS, or bare-metal).
  • A ServiceAccount, kubeconfig, or IAM role with read permissions on daemonsets.
  • A GitHub Personal Access Token with repository issue creation permissions.
  • A Slack incoming webhook URL for notifications.

Secrets

  • KUBECONFIG (or in-cluster ServiceAccount): Authentication credentials for Kubernetes API.
  • GITHUB_TOKEN: GitHub token used to open incident issues.
  • SLACK_WEBHOOK: Incoming Webhook URL for cluster alerts.

Inputs

  • namespace (STRING, default ""): Optional namespace filter (leave empty to scan cluster-wide).
  • target_repository (STRING, default my-org/kubernetes-cluster-operations): Target GitHub repository for logging incident issues.

Outputs

  • outputs.probe_daemonset_status.vars.total_inspected: Total number of DaemonSets audited.
  • outputs.probe_daemonset_status.vars.deficit_count: Count of DaemonSets experiencing scheduling deficits.
  • outputs.probe_daemonset_status.vars.degraded_daemonsets: List of degraded DaemonSet objects (namespace, name, desired, ready, unavailable).

Quick start

  1. Configure GITHUB_TOKEN and SLACK_WEBHOOK in your Kestra namespace.
  2. Import this flow and set target_repository to match your operations tracker.
  3. Run the flow manually to verify DaemonSet status extraction and Slack notification delivery.
  4. Enable the periodic_daemonset_check trigger for continuous scheduling health monitoring.

How to extend

  • Add an automated node drain or uncordon task to clear blocked nodes.
  • Filter DaemonSets by critical labels (e.g., k8s-app=kube-dns) to prioritize networking components.
  • Route urgent alerts to PagerDuty or Opsgenie if system-critical DaemonSets drop below 80% node coverage.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.