Schedule icon
Script icon
If icon
Create icon
SlackIncomingWebhook icon
Log icon

GitHub Self-Hosted Runner Fleet Offline Sentinel

Monitor GitHub self-hosted runner health with Kestra. Automatically detect offline CI/CD nodes, create GitHub issues, and alert DevOps on Slack.

Categories
CoreInfrastructure

Enterprise software engineering teams rely on self-hosted GitHub Actions runners hosted across private AWS EC2 instances, GCP VMs, or Kubernetes clusters. When runner processes crash, disk volumes fill up, or host machines lose network reachability, jobs queue indefinitely and block continuous delivery pipelines. This blueprint queries GitHub's Actions API hourly for organization self-hosted runners, detects nodes in an offline status, files a prioritized GitHub tracking issue with full node metadata, and alerts on-call DevOps engineers via Slack.

How it works

  1. check_runner_health_hourly (io.kestra.plugin.core.trigger.Schedule) initiates the audit every hour (shipped disabled for safe verification).
  2. probe_offline_runners (io.kestra.plugin.scripts.python.Script) interrogates the GitHub Actions REST API (GET /orgs/{org}/actions/runners) and extracts nodes with status == 'offline'.
  3. branch_on_offline_runners (io.kestra.plugin.core.flow.If) evaluates whether any nodes are offline (has_offline == true):
    • then: Automatically files a GitHub tracking issue via create_runner_incident_issue (io.kestra.plugin.github.issues.Create) and alerts the DevOps Slack channel via alert_devops_slack (io.kestra.plugin.slack.notifications.SlackIncomingWebhook).
    • else: Emits an informational log entry (log_fleet_healthy) confirming full fleet availability.
  4. log_fleet_summary (io.kestra.plugin.core.log.Log) logs the final node counts.
  5. The errors block alerts Slack if API tokens expire or GitHub API limits are reached.

What you get

  • Automated, scheduled auditing of self-hosted CI/CD infrastructure.
  • Rapid detection of silent runner process crashes before developers report blocked PR builds.
  • Consolidated GitHub incident tickets with node names, operating systems, and runner IDs.
  • Immediate Slack notifications with direct ticket hyperlinks.

Who it's for

  • Platform Engineers and DevOps teams maintaining private CI/CD infrastructure.
  • Site Reliability Engineers safeguarding developer productivity and deployment throughput.
  • Engineering Managers monitoring continuous integration stability.

Why orchestrate this with Kestra

Kestra coordinates external API inspection, data filtering, conditional escalation, and chat notifications inside a unified, declarative pipeline. It injects personal access tokens securely, handles transient API failures, and maintains full execution logs without maintaining custom cron scripts or serverless bots.

Prerequisites

  • A GitHub Organization with self-hosted runners configured under Organization Settings.
  • A GitHub Personal Access Token or GitHub App token with admin:org (read runners) and repo permissions.
  • A Slack incoming webhook URL for notifications.

Secrets

  • GITHUB_TOKEN: Token used to query GitHub Actions runner APIs and open tracking issues.
  • SLACK_WEBHOOK: Incoming Webhook URL for platform alerts.

Inputs

  • github_organization (STRING, default my-enterprise-org): Name of the GitHub organization hosting the self-hosted runner pool.
  • target_repository (STRING, default my-enterprise-org/platform-engineering): Target GitHub repository for logging runner downtime issues.

Outputs

  • outputs.probe_offline_runners.vars.total_runners: Total number of registered runners in the organization.
  • outputs.probe_offline_runners.vars.offline_count: Count of runners currently in offline state.
  • outputs.probe_offline_runners.vars.offline_runners: List of offline runner objects (id, name, os, status, busy).

Quick start

  1. Configure the GITHUB_TOKEN and SLACK_WEBHOOK secrets in your Kestra namespace.
  2. Import this flow and set github_organization and target_repository to match your organization.
  3. Run the flow manually to verify runner status querying and Slack notification delivery.
  4. Enable the check_runner_health_hourly trigger for continuous fleet health monitoring.

How to extend

  • Add an automated AWS Auto Scaling or Kubernetes pod restart task to automatically remediate downed nodes.
  • Route critical alerts to PagerDuty or Opsgenie if more than 50% of the fleet drops offline.
  • Filter runners by labels (e.g., production-gpu) to prioritize mission-critical build infrastructure.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.