GitLab Runner Concurrent Job Capacity Sentinel Blueprint

id: gitlab-runner-concurrent-job-capacity-sentinel
namespace: company.cicd

description: |
  Audits self-hosted GitLab Runners for concurrent job saturation, offline executor states,
  and build queue delays to ensure continuous delivery pipeline capacity.

concurrency:
  limit: 1

inputs:
  - id: gitlab_api_endpoint
    type: STRING
    displayName: GitLab API Endpoint
    defaults: "https://gitlab.com/api/v4"
    description: Base REST API v4 endpoint for GitLab SaaS or self-hosted GitLab instance.

  - id: gitlab_runner_id
    type: INT
    displayName: GitLab Runner ID
    defaults: 123456
    description: Numeric identifier of the target self-hosted or fleet GitLab Runner.

  - id: max_concurrent_jobs_threshold
    type: INT
    displayName: Max Concurrent Jobs Threshold
    defaults: 10
    description: Maximum allowable running jobs on the runner before declaring
      capacity saturation.

  - id: alert_on_saturation
    type: BOOL
    displayName: Alert on Saturation
    defaults: true
    description: Whether to dispatch automated Slack alerts when runner capacity is
      saturated or offline.

tasks:
  - id: evaluate_gitlab_runner_capacity
    type: io.kestra.plugin.scripts.python.Script
    description: Queries GitLab REST API v4 for runner operational status and
      currently executing jobs.
    containerImage: python:3.11-slim
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
    beforeCommands:
      - pip install requests kestra
    env:
      GITLAB_TOKEN: "{{ secret('GITLAB_TOKEN') }}"
    script: |
      import os
      import json
      import requests
      from kestra import Kestra

      endpoint = "{{ inputs.gitlab_api_endpoint }}".rstrip('/')
      runner_id = int("{{ inputs.gitlab_runner_id }}")
      token = os.environ.get("GITLAB_TOKEN", "")
      max_jobs_threshold = int("{{ inputs.max_concurrent_jobs_threshold }}")

      headers = {
          "PRIVATE-TOKEN": token,
          "Accept": "application/json"
      }

      runner_description = f"Runner-{runner_id}"
      runner_status = "unknown"
      runner_online = False
      running_jobs = []
      is_saturated = False

      try:
          resp = requests.get(f"{endpoint}/runners/{runner_id}", headers=headers, timeout=15)
          if resp.status_code == 200:
              r_data = resp.json()
              runner_description = r_data.get("description", runner_description)
              runner_status = r_data.get("status", "unknown")
              runner_online = r_data.get("online", False) or runner_status == "online"

          jobs_resp = requests.get(f"{endpoint}/runners/{runner_id}/jobs?status=running", headers=headers, timeout=15)
          if jobs_resp.status_code == 200:
              jobs_data = jobs_resp.json()
              if isinstance(jobs_data, list):
                  for j in jobs_data:
                      running_jobs.append({
                          "id": j.get("id"),
                          "project": j.get("project", {}).get("name_with_namespace"),
                          "stage": j.get("stage"),
                          "name": j.get("name"),
                          "created_at": j.get("created_at")
                      })

          active_jobs_count = len(running_jobs)

          if (not runner_online) or (active_jobs_count >= max_jobs_threshold):
              is_saturated = True

          Kestra.outputs({
              "runner_id": runner_id,
              "runner_description": runner_description,
              "runner_status": runner_status,
              "runner_online": runner_online,
              "active_jobs_count": active_jobs_count,
              "is_saturated": is_saturated,
              "running_jobs_details": json.dumps(running_jobs[:10])
          })

      except Exception as e:
          print(f"Error querying GitLab Runner API: {e}")
          Kestra.outputs({
              "runner_id": runner_id,
              "runner_description": runner_description,
              "runner_status": "error",
              "runner_online": False,
              "active_jobs_count": 0,
              "is_saturated": True,
              "running_jobs_details": "[]"
          })

  - id: check_runner_saturation
    type: io.kestra.plugin.core.flow.If
    description: Evaluates whether GitLab runner is offline or active jobs exceed
      concurrency limits.
    condition: "{{ outputs.evaluate_gitlab_runner_capacity.vars.is_saturated and
      inputs.alert_on_saturation }}"
    then:
      - id: notify_slack_runner_saturation
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        description: Dispatches high-priority CI/CD runner capacity exhaustion alert to
          DevOps on Slack.
        url: "{{ secret('SLACK_WEBHOOK_URL') }}"
        payload: |
          {
            "channel": "#cicd-operations",
            "username": "GitLab-Runner-Sentinel",
            "text": "WARNING: GitLab Runner `{{ outputs.evaluate_gitlab_runner_capacity.vars.runner_description }}` (#{{ outputs.evaluate_gitlab_runner_capacity.vars.runner_id }}) is saturated or offline! Status: `{{ outputs.evaluate_gitlab_runner_capacity.vars.runner_status }}` (Online: {{ outputs.evaluate_gitlab_runner_capacity.vars.runner_online }}), Active Jobs: `{{ outputs.evaluate_gitlab_runner_capacity.vars.active_jobs_count }}` (Limit: {{ inputs.max_concurrent_jobs_threshold }}). GitLab Admin: {{ inputs.gitlab_api_endpoint }}/admin/runners"
          }

  - id: log_runner_capacity_status
    type: io.kestra.plugin.core.log.Log
    description: Records an operational audit trail of GitLab runner capacity evaluations.
    message: "GitLab Runner audit completed for '{{
      outputs.evaluate_gitlab_runner_capacity.vars.runner_description }}'.
      Status: {{ outputs.evaluate_gitlab_runner_capacity.vars.runner_status }},
      Active Jobs: {{
      outputs.evaluate_gitlab_runner_capacity.vars.active_jobs_count }},
      Saturated: {{ outputs.evaluate_gitlab_runner_capacity.vars.is_saturated
      }}."

  - id: export_gitlab_manifest
    type: io.kestra.plugin.core.debug.Return
    description: Emits structured manifest of GitLab runner capacity for CI/CD
      analytics and autoscaling controllers.
    format:
      runner_id: "{{ outputs.evaluate_gitlab_runner_capacity.vars.runner_id }}"
      runner_description: "{{ outputs.evaluate_gitlab_runner_capacity.vars.runner_description }}"
      status: "{{ outputs.evaluate_gitlab_runner_capacity.vars.runner_status }}"
      online: "{{ outputs.evaluate_gitlab_runner_capacity.vars.runner_online }}"
      active_jobs: "{{ outputs.evaluate_gitlab_runner_capacity.vars.active_jobs_count }}"
      is_saturated: "{{ outputs.evaluate_gitlab_runner_capacity.vars.is_saturated }}"
      jobs_sample: "{{ outputs.evaluate_gitlab_runner_capacity.vars.running_jobs_details }}"

triggers:
  - id: ten_minute_gitlab_audit
    type: io.kestra.plugin.core.trigger.Schedule
    description: Automated schedule trigger auditing GitLab runner capacity every
      ten minutes.
    cron: "*/10 * * * *"

Modern software development teams depend on GitLab CI/CD pipelines for automated testing, container builds, and production Kubernetes deployments. High-velocity engineering teams frequently operate private, self-hosted GitLab Runners on AWS EC2, GCP Compute Engine, or dedicated bare-metal servers to leverage specialized hardware such as GPUs or to access internal VPC subnets. However, during morning commit surges or scheduled release trains, self-hosted runner executors reach concurrency limits. As a consequence, builds queue silently in pending states, developer pull request turnaround times spike, and time-to-market degrades. If a runner daemon crashes or Docker daemon socket exhaustion occurs, pipelines stall indefinitely without explicit failure alerts. This sentinel polls the GitLab REST API v4 every 10 minutes, verifies executor online states, counts active in-flight jobs, and dispatches incident notifications to DevOps teams on Slack before developer build queues back up.

How it works

Scheduled triggering: Executes every 10 minutes via automated cron trigger (*/10 * * * *) or runs on-demand via manual execution.
Runner status retrieval: Queries the GitLab REST API endpoint /runners/{runner_id} to verify connectivity and retrieve executor operational status.
Active job enumeration: Calls /runners/{runner_id}/jobs?status=running to obtain the list of currently executing pipeline builds.
Capacity threshold evaluation: Compares active in-flight jobs against max_concurrent_jobs_threshold and verifies that the runner is marked online.
Conditional alerting: An io.kestra.plugin.core.flow.If flowable checks if the runner is saturated or offline and dispatches a Slack alert to #cicd-operations.
Manifest export: Emits a structured JSON audit payload detailing active jobs and runner status for downstream autoscaling or capacity planning.

Architecture diagram

flowchart TD
  A[ten_minute_gitlab_audit Schedule Trigger] --> B[evaluate_gitlab_runner_capacity Python Script]
  B --> C[check_runner_saturation Flowable If]
  C -- Runner Saturated or Offline --> D[notify_slack_runner_saturation Webhook]
  C -- Capacity Nominal --> E[log_runner_capacity_status Log]
  D --> E
  E --> F[export_gitlab_manifest Return]

What you get

Proactive detection of bottlenecked CI/CD runners before developer pull requests encounter long queue delays.
Immediate alerting when a self-hosted runner daemon goes offline, uncontactable, or paused.
Contextual Slack alerts detailing active jobs, project repositories, and direct GitLab admin console links.
Structured JSON execution metrics to drive horizontal runner autoscaling automation.

Who it's for

DevOps and Platform Engineers managing private Kubernetes or VM-based GitLab Runner fleets.
Release Managers ensuring timely build, test, and container deployment cycles across engineering.
Engineering Managers monitoring CI/CD pipeline infrastructure health and developer productivity metrics.

Why orchestrate this with Kestra

GitLab CI pipelines queue silently when runners hit concurrency ceilings or underlying Docker daemons lock up. Kestra provides an independent, out-of-band control loop that interrogates the GitLab API, validates executor health, and coordinates alerting or auto-remediation workflows. With Kestra's containerized Python runner and native secret management, platform teams can manage fleet reliability without writing brittle custom daemons.

Use cases

Self-Hosted Runner Availability Watchdog: Detect when a private runner host encounters an unexpected kernel panic or Docker socket failure.
CI/CD Queue Bottleneck Prevention: Identify when concurrent job allocations approach hardware limits and trigger additional runner instances.
Multi-Project Shared Runner Governance: Monitor shared runner fleets across multiple teams to prevent noisy-neighbor build monopolization.

Inputs

Name	Type	Default	Description
`gitlab_api_endpoint`	`STRING`	`https://gitlab.com/api/v4`	Base REST API v4 endpoint for GitLab SaaS or self-hosted GitLab instance.
`gitlab_runner_id`	`INT`	`123456`	Numeric identifier of the target self-hosted or fleet GitLab Runner.
`max_concurrent_jobs_threshold`	`INT`	`10`	Maximum allowable running jobs on the runner before declaring capacity saturation.
`alert_on_saturation`	`BOOL`	`true`	Whether to dispatch automated Slack alerts when runner capacity is saturated or offline.

Expected outputs

{{ outputs.evaluate_gitlab_runner_capacity.vars.runner_description }}: Display name or description of the evaluated GitLab runner.
{{ outputs.evaluate_gitlab_runner_capacity.vars.active_jobs_count }}: Number of currently running CI/CD jobs on the runner.
{{ outputs.evaluate_gitlab_runner_capacity.vars.is_saturated }}: Boolean flag indicating whether the runner has breached capacity limits or gone offline.
{{ outputs.export_gitlab_manifest.value }}: Structured dictionary containing the consolidated runner health summary.

Prerequisites

Outbound HTTPS network access from Kestra execution workers to the GitLab API endpoint.
A GitLab Personal Access Token or Project Token with read_api scope.
A Slack Incoming Webhook URL configured in Kestra secrets.

Secrets

GITLAB_TOKEN: GitLab access token with permissions to read runner configurations and active jobs.
SLACK_WEBHOOK_URL: Slack Incoming Webhook URL used to dispatch CI/CD incident notifications.

Quick start

Register GITLAB_TOKEN and SLACK_WEBHOOK_URL in your Kestra secret storage.
Import this blueprint into your Kestra instance under the company.cicd namespace.
Update gitlab_runner_id with your runner ID (found under GitLab Project Settings > CI/CD > Runners).
Run a manual execution to verify runner discovery and validate the diagnostic output format.

Common pitfalls and troubleshooting

Token Permissions: Ensure the token possesses read_api or admin:read scope; standard repository tokens without API permissions will return 401 Unauthorized.
Self-Hosted TLS Certificates: For self-hosted GitLab installations with private CAs, ensure your Kestra worker environment trusts the internal CA root.
Runner Registration States: If a runner is marked paused in the GitLab UI, the sentinel flags it as saturated to prevent silent job accumulation.

How to extend

Automated Runner Autoscaling: Trigger an AWS Auto Scaling Group or Kubernetes HPA scale-out event when concurrent jobs exceed 80% capacity.
Docker Daemon Pruning: Dispatch an SSH or Ansible command to restart gitlab-runner and execute docker system prune if executor errors occur.
Fleet-Wide Matrix Scanning: Wrap the evaluation in an io.kestra.plugin.core.flow.ForEach loop to monitor all runners registered across the GitLab organization.

GitLab Runner Concurrent Job Capacity Sentinel

How it works

Architecture diagram

What you get

Who it's for

Why orchestrate this with Kestra

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

GitLab Runner Concurrent Job Capacity Sentinel

How it works

Architecture diagram

What you get

Who it's for

Why orchestrate this with Kestra

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

DigitalOcean Alert Policy Compliance Gate

DigitalOcean Account Balance Depletion Sentinel

Dex IdP SAML Connector Authentication Error Sentinel

DigitalOcean Stale Snapshot Cost Sentinel

Nightly DigitalOcean Droplet Snapshot Backups by Tag

DigitalOcean New Droplet Governance Guard

More Core Blueprints

Block breaking OpenAPI changes before they merge

Error handling: send Slack alert on failure

Gate OpenFGA Model Changes Against Real Authorization Decisions

Open Graph Card Generator

Open a GitHub issue only for vulnerabilities that appeared since the last scan

Add a parametrized Python script as a Namespace File and run it in

New to Kestra?