Schedule icon
Request icon
Script icon
Set icon
Switch icon
SlackIncomingWebhook icon
PagerDutyAlert icon
Fail icon
Log icon

Tailscale Subnet Router Failover & High-Availability Sentinel

Monitor Tailscale HA subnet routers, verify route approvals, detect router failover, and alert via Slack and PagerDuty on VPC connectivity outages.

Categories
CloudInfrastructure

Tailscale Subnet Routers act as zero-trust gateways, bridging Tailscale overlay networks with private clouds, VPCs, and on-premises subnets. Production architectures configure high-availability (HA) router pairs where primary and standby nodes advertise identical CIDR routes. If the primary subnet router fails, Tailscale clients automatically route traffic through the standby node, maintaining VPC connectivity.

However, silent router failures or unapproved routes jeopardize high availability. If the primary router goes offline and no healthy standby exists, cross-network traffic stops immediately. This blueprint monitors Tailscale subnet routers, verifies advertised and enabled CIDR routes, detects node outages, evaluates failover state, and dispatches automated alerts to Slack and PagerDuty before network downtime impacts services.

How it works

  1. Scheduled Audit Trigger: Runs on a configurable 5-minute schedule (*/5 * * * *), disabled by default for controlled onboarding and manual execution.
  2. Device Inventory Probe: Issues an HTTP GET request to https://api.tailscale.com/api/v2/tailnet/{tailnet}/devices?fields=all with Bearer authentication to retrieve tailnet nodes, routing configurations, and connection telemetry.
  3. Route & HA Failover Audit: A Python script inspects nodes advertising the target subnet CIDR (advertisedRoutes vs enabledRoutes), checks node connectivity (online status and lastSeen heartbeat age), and identifies designated primary and standby routers.
  4. State Classification: Categorizes tailnet subnet routing state into NOMINAL_HEALTHY, HA_FAILOVER_ACTIVE, STANDBY_ROUTER_DEGRADED, ROUTE_APPROVAL_PENDING, or TOTAL_SUBNET_OUTAGE.
  5. Stateful KV Storage: Records the full subnet health audit, active routing node, and redundancy metrics in the Kestra Key-Value store (tailscale_subnet_10_0_0_0_16) for continuous operational visibility.
  6. Multi-Channel Incident Dispatch: Dispatches structured Slack messages and escalates total connectivity loss incidents to PagerDuty.
  7. Deterministic Failure Execution: Terminates degraded or outage executions with Fail task to trigger SLA breach notifications.

What you get

  • Continuous HA Failover Detection: Instant detection when traffic fails over from primary to standby subnet routers.
  • Route Approval Auditing: Verification that advertised CIDR prefixes are explicitly approved in the Tailscale admin console.
  • Total Outage Safeguards: Immediate escalation via Slack and PagerDuty when all routers for a subnet go offline.
  • Persistent Health Telemetry: Subnet connectivity status and active node metrics persisted in Kestra KV store.
  • Declarative Network Sentinel: Version-controlled YAML pipeline replacing unmonitored scripts and manual checks.

Who it's for

  • Network & Cloud Engineers: Managing hybrid cloud connectivity, site-to-site meshes, and VPC peering.
  • Platform & DevOps Teams: Running container workloads and database connections over Tailscale overlay networks.
  • Site Reliability Engineers: Ensuring 99.99% availability for critical cross-network transit routes.

Why orchestrate this with Kestra

  • Native Secret Management: Injects Tailscale API keys, Slack webhooks, and PagerDuty routing keys securely without disk leakage.
  • Unified Logic Execution: Combines HTTP REST probes, lightweight Python analytics, and built-in KV storage in a single flow.
  • Deterministic Alert Routing: Uses Switch task branching to decouple nominal logging from emergency multi-channel paging.
  • Audit History: Retains detailed execution logs and historical health transitions for post-incident audits.

Prerequisites

  • A Tailscale tailnet with at least one subnet router configured and advertising a CIDR route.
  • A Tailscale API access token or OAuth client credential with devices:read permissions.
  • Network access from Kestra workers to https://api.tailscale.com.
  • A Slack incoming webhook URL for alert notifications.
  • (Optional) A PagerDuty Events API v2 routing key for incident escalation.

Secrets

  • TAILSCALE_API_KEY: Tailscale API access token or OAuth token with permissions to query devices.
  • SLACK_WEBHOOK_URL: Slack Incoming Webhook URL for incident notifications.
  • PAGERDUTY_ROUTING_KEY: PagerDuty Events API v2 routing key (required only if enable_pagerduty is true).

Quick start

  1. Create required secrets TAILSCALE_API_KEY and SLACK_WEBHOOK_URL in your Kestra namespace.
  2. If using PagerDuty, create PAGERDUTY_ROUTING_KEY and set enable_pagerduty to true.
  3. Set tailnet to your Tailscale organization domain or - for default tailnet.
  4. Set target_subnet to your target VPC CIDR (e.g., 10.0.0.0/16).
  5. Set primary_router_hostname to your primary subnet router hostname (e.g., subnet-router-primary).
  6. Run the flow manually in Kestra UI to verify discovery of primary and standby routers.
  7. Enable the schedule trigger to establish automated 5-minute continuous monitoring.

How to extend

  • Automated Router Restart: Add tasks to restart AWS EC2 or GCP Compute instances hosting the failed primary router via cloud provider plugins.
  • Dynamic ACL Updates: Call Tailscale ACL API endpoints to approve new standby routers automatically during auto-scaling events.
  • Multi-Region Mesh Monitoring: Chain multiple subnets into parallel subflows to monitor multi-VPC topologies simultaneously.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.