Headscale Tailnet Node Health & Expiry Sentinel

id: headscale-tailnet-node-health-and-expiry-sentinel
namespace: company.team

description: |
  Audits Headscale self-hosted Tailscale control planes for node connectivity, evaluates machine key expiration deadlines, classifies tailnet mesh health, and routes incident alerts to Slack.

triggers:
  - id: daily_morning_audit
    type: io.kestra.plugin.core.trigger.Schedule
    description: Executes automated Headscale tailnet node and key expiry audits
      daily at 08:00 AM.
    cron: "0 8 * * *"
    disabled: true
    inputs:
      headscale_host: "headscale"
      headscale_port: 8080
      days_to_expiry_threshold: 14
      alert_on_offline_nodes: "true"

inputs:
  - id: headscale_host
    type: STRING
    displayName: Headscale Hostname
    description: Hostname or IP address of the Headscale control plane server
      reachable by the Kestra worker.
    defaults: "headscale"

  - id: headscale_port
    type: INT
    displayName: Headscale HTTP Port
    description: HTTP port of the Headscale API and health service (defaults to 8080).
    defaults: 8080

  - id: days_to_expiry_threshold
    type: INT
    displayName: Key Expiry Warning Threshold (Days)
    description: Number of days prior to machine key expiration at or below which an
      alert is triggered.
    defaults: 14

  - id: alert_on_offline_nodes
    type: SELECT
    displayName: Alert on Offline Nodes
    description: Whether to raise a WARNING alert when registered tailnet nodes are
      disconnected or offline.
    values:
      - "true"
      - "false"
    defaults: "true"

tasks:
  - id: probe_headscale_health
    type: io.kestra.plugin.core.http.Request
    description: Queries the Headscale health endpoint to verify control plane
      process responsiveness.
    uri: "http://{{ inputs.headscale_host }}:{{ inputs.headscale_port }}/health"
    method: GET
    options:
      allowFailed: true
    allowFailure: true

  - id: probe_headscale_nodes
    type: io.kestra.plugin.core.http.Request
    description: Queries registered tailnet nodes to inspect connection status,
      assigned IP addresses, and key expiry timestamps.
    uri: "http://{{ inputs.headscale_host }}:{{ inputs.headscale_port
      }}/api/v1/node"
    method: GET
    headers:
      Authorization: "Bearer {{ secret('HEADSCALE_API_KEY') }}"
      Accept: "application/json"
    options:
      allowFailed: true
    allowFailure: true

  - id: probe_headscale_users
    type: io.kestra.plugin.core.http.Request
    description: Queries registered Headscale users and namespaces to validate
      multi-tenant directory vitality.
    uri: "http://{{ inputs.headscale_host }}:{{ inputs.headscale_port
      }}/api/v1/user"
    method: GET
    headers:
      Authorization: "Bearer {{ secret('HEADSCALE_API_KEY') }}"
      Accept: "application/json"
    options:
      allowFailed: true
    allowFailure: true

  - id: classify_tailnet_status
    type: io.kestra.plugin.core.debug.Return
    description: Evaluates API response codes, node expiration timestamps, and
      connectivity to classify tailnet state into HEALTHY,
      OFFLINE_NODES_WARNING, EXPIRED_NODES_CRITICAL, PROBE_FAILED, or
      CONTROL_PLANE_UNREACHABLE.
    format: |
      {{ (outputs.probe_headscale_health.code is not defined or outputs.probe_headscale_health.code != 200) ? 'CONTROL_PLANE_UNREACHABLE' : ( (outputs.probe_headscale_nodes.code is not defined or outputs.probe_headscale_nodes.code != 200) ? 'PROBE_FAILED' : ( ((outputs.probe_headscale_nodes.body | json | jq('[.nodes[]? | select((.expiry != null and .expiry != "" and .expiry != "0001-01-01T00:00:00Z") and (((.expiry | fromdateiso8601) - now) / 86400) < ' ~ inputs.days_to_expiry_threshold ~ ')] | length') | first) > 0) ? 'EXPIRED_NODES_CRITICAL' : ( (inputs.alert_on_offline_nodes == 'true' and ((outputs.probe_headscale_nodes.body | json | jq('[.nodes[]? | select(.online == false)] | length') | first) > 0)) ? 'OFFLINE_NODES_WARNING' : 'HEALTHY' ) ) ) }}

  - id: route_by_tailnet_status
    type: io.kestra.plugin.core.flow.Switch
    description: Routes workflow notifications based on the classified tailnet mesh status.
    value: "{{ outputs.classify_tailnet_status.value }}"
    cases:
      "HEALTHY":
        - id: log_healthy_state
          type: io.kestra.plugin.core.log.Log
          message: "Headscale tailnet on {{ inputs.headscale_host }}:{{
            inputs.headscale_port }} is HEALTHY. Registered nodes: {{
            outputs.probe_headscale_nodes.body | json | jq('.nodes | length') |
            first }}. Active users: {{ outputs.probe_headscale_users.body | json
            | jq('.users | length') | first }}. All machine keys valid."

      "OFFLINE_NODES_WARNING":
        - id: alert_offline_nodes_slack
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          url: "{{ secret('SLACK_WEBHOOK') }}"
          payload: |
            {{ {"text": "Headscale Tailnet WARNING on " ~ inputs.headscale_host ~ ":" ~ inputs.headscale_port ~ ". Detected offline node(s): " ~ (outputs.probe_headscale_nodes.body | json | jq('[.nodes[]? | select(.online == false) | .givenName + \" (\" + (.ipAddresses[0] // \"no-ip\") + \")\"] | join(\", \")') | first) ~ ". Execution: " ~ execution.id ~ "."} | toJson }}

        - id: log_offline_warning
          type: io.kestra.plugin.core.log.Log
          message: "Offline nodes detected: {{ outputs.probe_headscale_nodes.body | json |
            jq('[.nodes[]? | select(.online == false) | .name] | join(\", \")')
            | first }}."

      "EXPIRED_NODES_CRITICAL":
        - id: alert_expired_nodes_slack
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          url: "{{ secret('SLACK_WEBHOOK') }}"
          payload: |
            {{ {"text": "Headscale Tailnet CRITICAL on " ~ inputs.headscale_host ~ ":" ~ inputs.headscale_port ~ ". Machine key expiring within " ~ inputs.days_to_expiry_threshold ~ " day(s): " ~ (outputs.probe_headscale_nodes.body | json | jq('[.nodes[]? | select((.expiry != null and .expiry != \"\" and .expiry != \"0001-01-01T00:00:00Z\") and (((.expiry | fromdateiso8601) - now) / 86400) < ' ~ inputs.days_to_expiry_threshold ~ ') | .givenName + \" (expires: \" + .expiry + \")\"] | join(\", \")') | first) ~ ". Immediate re-authentication required. Execution: " ~ execution.id ~ "."} | toJson }}

      "PROBE_FAILED":
        - id: alert_probe_failed_slack
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          url: "{{ secret('SLACK_WEBHOOK') }}"
          payload: |
            {{ {"text": "Headscale Sentinel PROBE_FAILED on " ~ inputs.headscale_host ~ ":" ~ inputs.headscale_port ~ ". Health check passed but node API returned HTTP " ~ outputs.probe_headscale_nodes.code ~ ". Check HEADSCALE_API_KEY permissions. Execution: " ~ execution.id ~ "."} | toJson }}

      "CONTROL_PLANE_UNREACHABLE":
        - id: alert_unreachable_slack
          type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
          url: "{{ secret('SLACK_WEBHOOK') }}"
          payload: |
            {{ {"text": "Headscale Sentinel CONTROL_PLANE_UNREACHABLE. Cannot reach control plane at http://" ~ inputs.headscale_host ~ ":" ~ inputs.headscale_port ~ "/health. Service is offline or network partitioned. Execution: " ~ execution.id ~ "."} | toJson }}

  - id: log_audit_summary
    type: io.kestra.plugin.core.output.OutputValues
    description: Exports structured telemetry attributes and tailnet node metrics
      for compliance records.
    values:
      tailnet_status: "{{ outputs.classify_tailnet_status.value }}"
      total_nodes: "{{ outputs.probe_headscale_nodes.body | json | jq('.nodes |
        length') | first ?? 0 }}"
      online_nodes_count: "{{ outputs.probe_headscale_nodes.body | json |
        jq('[.nodes[]? | select(.online == true)] | length') | first ?? 0 }}"
      offline_nodes_count: "{{ outputs.probe_headscale_nodes.body | json |
        jq('[.nodes[]? | select(.online == false)] | length') | first ?? 0 }}"
      expiring_nodes_count: "{{ outputs.probe_headscale_nodes.body | json |
        jq('[.nodes[]? | select((.expiry != null and .expiry != \"\" and .expiry
        != \"0001-01-01T00:00:00Z\") and (((.expiry | fromdateiso8601) - now) /
        86400) < ' ~ inputs.days_to_expiry_threshold ~ ')] | length') | first ??
        0 }}"

errors:
  - id: alert_flow_error
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    description: Dispatches a high-priority Slack notification if an unhandled error
      halts sentinel execution.
    url: "{{ secret('SLACK_WEBHOOK') }}"
    payload: |
      {{ {"text": "Headscale Sentinel workflow error during execution " ~ execution.id ~ " in namespace " ~ flow.namespace ~ "."} | toJson }}

Continuously audits Headscale self-hosted Tailscale control planes by querying REST APIs, monitoring WireGuard mesh node connectivity, detecting impending machine key expiration deadlines, and routing escalation alerts to Slack.

How it works

probe_headscale_health checks /health on the Headscale control plane server to verify daemon process vitality and HTTP API responsiveness.
probe_headscale_nodes queries /api/v1/node using a Bearer token to inspect all enrolled tailnet machines, IP addresses, connectivity statuses, and key expiration dates.
probe_headscale_users queries /api/v1/user to verify registered namespace accounts and multi-tenant domain integrity.
classify_tailnet_status evaluates node attributes through a Pebble expression:
- Flags machines whose authentication keys expire within days_to_expiry_threshold and categorizes the state as EXPIRED_NODES_CRITICAL.
- Flags disconnected nodes (online: false) when alert_on_offline_nodes is active and categorizes the state as OFFLINE_NODES_WARNING.
- Confirms all enrolled nodes are connected with valid keys to categorize the state as HEALTHY.
- Detects unreachable endpoints or non-200 responses to classify state as CONTROL_PLANE_UNREACHABLE or PROBE_FAILED.
route_by_tailnet_status branches workflow execution based on the classification:
- HEALTHY: Logs active node tallies, registered users, and key health status.
- OFFLINE_NODES_WARNING: Dispatches a Slack notification naming disconnected tailnet nodes and assigned IP addresses.
- EXPIRED_NODES_CRITICAL: Posts an urgent escalation alert to Slack naming machines approaching key expiration before mesh connectivity drops.
- CONTROL_PLANE_UNREACHABLE: Alerts operations that the Headscale server is completely unreachable.
log_audit_summary produces structured outputs recording node tallies, online/offline ratios, and expiration metrics for security auditing.
errors catches unexpected task exceptions and posts a diagnostic notice to Slack.

What you get

Automated, continuous surveillance of self-hosted Tailscale overlay mesh networks without third-party monitoring agents.
Early warning before node keys expire, preventing sudden tunnel collapse and communication outages across databases, CI runners, and internal servers.
Immediate Slack alerts detailing machine hostnames, given names, assigned Tailscale IPs, and expiry timestamps.
Centralized audit tracking of network mesh health, node capacity, and user counts.
Automated scheduled daily auditing coupled with manual on-demand execution support.

Who it's for

Platform Engineers and Site Reliability Engineers managing self-hosted Headscale / Tailscale mesh infrastructure.
Network Administrators and Security Engineers operating zero-trust overlay networks for internal service communication.
DevSecOps teams responsible for maintaining high availability across distributed hybrid-cloud and edge nodes.

Why orchestrate this with Kestra

Headscale manages zero-trust network connectivity between critical infrastructure components. When an enrolled machine's authentication key expires, WireGuard tunnels silently drop, severing database replication, monitoring, and administrative access. Kestra provides declarative, reproducible orchestration with built-in scheduling, centralized secret management, granular failure classification, and auditable history across all network peers.

Prerequisites

A running Headscale control plane instance with HTTP port 8080 accessible from the Kestra worker.
A Headscale API key generated using headscale apikeys create.
An incoming Slack webhook URL for alert delivery.

Secrets

HEADSCALE_API_KEY: API token used for Bearer authentication against Headscale /api/v1/ endpoints.
SLACK_WEBHOOK: Incoming Slack webhook URL used by alert_offline_nodes_slack, alert_expired_nodes_slack, alert_probe_failed_slack, alert_unreachable_slack, and the errors handler.

Quick start

Generate an API key on your Headscale instance:

headscale apikeys create --expiration 90d

Configure HEADSCALE_API_KEY and SLACK_WEBHOOK as secrets in your Kestra namespace.

Start a local Headscale test instance using Docker:

docker run -d --name headscale -p 8080:8080 headscale/headscale:latest headscale serve

Execute this flow manually in Kestra setting headscale_host: "localhost" or "headscale".
The workflow queries enrolled nodes, verifies key expiry, classifies HEALTHY, and records audit outputs.
Enable the daily_morning_audit trigger to automate continuous daily morning monitoring.

How to extend

Add automated key renewal commands using the Kestra Docker or SSH plugin to re-authenticate expiring nodes.
Integrate with PagerDuty for EXPIRED_NODES_CRITICAL states affecting production database primary or replica nodes.
Route separate alerts to Slack channels partitioned by Headscale user namespaces.
Track node IP allocation drift against IPAM records by integrating with the Kestra NetBox or phpIPAM plugins.

Headscale Tailnet Node Health and Expiry Sentinel

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Headscale Tailnet Node Health and Expiry Sentinel

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

Autonomous React Component Generator with Deep Learning Visual QA

AI-Powered Incident Response Automation

AI Incident Post-Mortem and Root Cause Generator

Autonomous Multi-Agent Software Engineering and Self-Healing Pipeline

AI Kubernetes Workload Hardening and Security Policy Advisor

AI-Powered Incident Triage & Automated Response

New to Kestra?