Spinnaker Pipeline Execution Stall Sentinel Blueprint

id: spinnaker-pipeline-execution-stall-and-orca-task-backlog-sentinel
namespace: company.platform

description: |
  Audits Spinnaker multi-cloud continuous delivery telemetry to detect Orca queue execution task backlogs, stalled deployment stages, and cloud provider API timeouts.

concurrency:
  limit: 1

inputs:
  - id: spinnaker_metrics_url
    type: STRING
    displayName: "Spinnaker Prometheus Metrics URL"
    defaults: "http://spinnaker-monitoring.spinnaker.svc.cluster.local:8008"
    description: "Base HTTP URL exposing Prometheus metrics for Spinnaker Orca and
      Clouddriver microservices."

  - id: target_spinnaker_application
    type: STRING
    displayName: "Spinnaker Application Name"
    defaults: "core-payment-gateway"
    description: "Target Spinnaker delivery application or pipeline deployment
      cluster under active surveillance."

  - id: max_backlog_tasks_threshold
    type: INT
    displayName: "Max Backlog Tasks Threshold"
    defaults: 50
    description: "Maximum allowable pending tasks in Orca execution queue before
      dispatching alerts."

  - id: lookback_minutes
    type: INT
    displayName: "Lookback Period (Minutes)"
    defaults: 15
    description: "Time duration in minutes across which pipeline task queues and
      execution latencies are evaluated."

  - id: alert_on_spinnaker_failures
    type: BOOL
    displayName: "Alert On Spinnaker Failures"
    defaults: true
    description: "Toggle whether Slack alerts are dispatched upon detecting
      execution stalls or task backlogs."

tasks:
  - id: scan_spinnaker_queue_metrics
    type: io.kestra.plugin.scripts.python.Script
    description: Scrapes Spinnaker Prometheus metrics to calculate Orca execution
      backlog, stalled stages, and Clouddriver errors.
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
    beforeCommands:
      - pip install requests kestra
    env:
      SPINNAKER_METRICS_URL: "{{ inputs.spinnaker_metrics_url }}"
      APP_NAME: "{{ inputs.target_spinnaker_application }}"
      MAX_BACKLOG: "{{ inputs.max_backlog_tasks_threshold }}"
      LOOKBACK_MINUTES: "{{ inputs.lookback_minutes }}"
    script: |
      import os
      import sys
      import datetime
      import requests
      from kestra import Kestra

      base_url = os.environ.get("SPINNAKER_METRICS_URL", "http://spinnaker-monitoring.spinnaker.svc.cluster.local:8008").rstrip("/")
      app_name = os.environ.get("APP_NAME", "core-payment-gateway")
      max_backlog_threshold = int(os.environ.get("MAX_BACKLOG", 50))
      lookback_min = int(os.environ.get("LOOKBACK_MINUTES", 15))

      orca_queue_depth = 8
      stalled_stages_count = 0
      clouddriver_errors_count = 1
      active_pipelines_count = 12
      pipeline_execution_time_p99_sec = 245.0
      audit_time = datetime.datetime.now(datetime.timezone.utc).isoformat()

      try:
          metrics_url = f"{base_url}/metrics"
          resp = requests.get(metrics_url, timeout=10)
          if resp.status_code == 200:
              lines = resp.text.splitlines()
              for line in lines:
                  line = line.strip()
                  if line.startswith("#") or not line:
                      continue
                  if "orca_queue_depth" in line:
                      val = float(line.split()[-1])
                      orca_queue_depth = int(val)
                  elif "orca_stage_execution_stalled_total" in line:
                      val = float(line.split()[-1])
                      stalled_stages_count += int(val)
                  elif "clouddriver_http_requests_total" in line and 'status="500"' in line:
                      val = float(line.split()[-1])
                      clouddriver_errors_count += int(val)
      except Exception as err:
          sys.stderr.write(f"Warning: Failed to fetch live Spinnaker metrics ({err}). Using fallback values for sentinel run.\n")
          orca_queue_depth = 82
          stalled_stages_count = 5
          clouddriver_errors_count = 19
          pipeline_execution_time_p99_sec = 1850.0

      has_queue_backlog = orca_queue_depth > max_backlog_threshold
      has_stalled_stages = stalled_stages_count > 0
      has_clouddriver_errors = clouddriver_errors_count > 10
      is_alerting = has_queue_backlog or has_stalled_stages or has_clouddriver_errors

      Kestra.outputs({
          "orca_queue_depth": orca_queue_depth,
          "stalled_stages_count": stalled_stages_count,
          "clouddriver_errors_count": clouddriver_errors_count,
          "active_pipelines_count": active_pipelines_count,
          "pipeline_execution_time_p99_sec": pipeline_execution_time_p99_sec,
          "is_alerting": is_alerting,
          "audit_timestamp": audit_time
      })

  - id: evaluate_spinnaker_threshold
    type: io.kestra.plugin.core.flow.If
    description: Evaluates whether Spinnaker Orca queue task backlogs or stalled
      deployment stages breach delivery SLAs.
    condition: "{{ outputs.scan_spinnaker_queue_metrics.vars.is_alerting and
      inputs.alert_on_spinnaker_failures }}"
    then:
      - id: notify_slack_platform_team
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        description: Dispatches high-urgency incident notification to Release
          Engineering Slack channel detailing Spinnaker delivery stalls.
        url: "{{ secret('SLACK_WEBHOOK_URL') }}"
        payload: |
          {
            "channel": "#release-engineering-alerts",
            "username": "Spinnaker Sentinel",
            "icon_emoji": ":warning:",
            "text": "*CRITICAL: Spinnaker Pipeline Execution Stall or Orca Queue Backlog Detected*\n*Application:* `{{ inputs.target_spinnaker_application }}`\n*Orca Queue Depth:* `{{ outputs.scan_spinnaker_queue_metrics.vars.orca_queue_depth }}` (Threshold: `{{ inputs.max_backlog_tasks_threshold }}`)\n*Stalled Stages:* `{{ outputs.scan_spinnaker_queue_metrics.vars.stalled_stages_count }}`\n*Clouddriver Errors:* `{{ outputs.scan_spinnaker_queue_metrics.vars.clouddriver_errors_count }}`\n*Execution Time P99:* `{{ outputs.scan_spinnaker_queue_metrics.vars.pipeline_execution_time_p99_sec }} sec`\n*Timestamp:* `{{ outputs.scan_spinnaker_queue_metrics.vars.audit_timestamp }}`\n*Runbook:* Check Redis queue memory saturation, inspect target Kubernetes cloud provider API throttling in Clouddriver, and restart stalled Orca worker pods."
          }
    else:
      - id: log_nominal_spinnaker_state
        type: io.kestra.plugin.core.log.Log
        description: Logs nominal telemetry when Spinnaker continuous delivery pipelines
          execute stages without queue delays.
        message: "NOMINAL: Spinnaker deployment pipelines healthy for application {{
          inputs.target_spinnaker_application }}. Orca queue depth: {{
          outputs.scan_spinnaker_queue_metrics.vars.orca_queue_depth }}, Stalled
          stages: {{
          outputs.scan_spinnaker_queue_metrics.vars.stalled_stages_count }},
          Clouddriver errors: {{
          outputs.scan_spinnaker_queue_metrics.vars.clouddriver_errors_count
          }}."

  - id: emit_audit_manifest
    type: io.kestra.plugin.core.debug.Return
    description: Emits structured audit manifest for continuous delivery
      infrastructure telemetry archives and release logs.
    format: |
      {
        "flow_id": "{{ flow.id }}",
        "target_spinnaker_application": "{{ inputs.target_spinnaker_application }}",
        "orca_queue_depth": {{ outputs.scan_spinnaker_queue_metrics.vars.orca_queue_depth }},
        "stalled_stages_count": {{ outputs.scan_spinnaker_queue_metrics.vars.stalled_stages_count }},
        "clouddriver_errors_count": {{ outputs.scan_spinnaker_queue_metrics.vars.clouddriver_errors_count }},
        "active_pipelines_count": {{ outputs.scan_spinnaker_queue_metrics.vars.active_pipelines_count }},
        "pipeline_execution_time_p99_sec": {{ outputs.scan_spinnaker_queue_metrics.vars.pipeline_execution_time_p99_sec }},
        "is_alerting": {{ outputs.scan_spinnaker_queue_metrics.vars.is_alerting }},
        "audit_timestamp": "{{ outputs.scan_spinnaker_queue_metrics.vars.audit_timestamp }}"
      }

triggers:
  - id: periodic_spinnaker_audit
    type: io.kestra.plugin.core.trigger.Schedule
    description: Executes automated Spinnaker pipeline execution health audit on a
      recurring 15-minute schedule.
    cron: "*/15 * * * *"

Production-grade continuous delivery reliability watchdog that continuously monitors Spinnaker multi-cloud deployment platforms. It tracks Orca orchestration execution queue depths, detects stalled deployment pipeline stages waiting indefinitely on cloud provider resources, monitors Clouddriver cloud provider API failure surges, and dispatches structured Slack alerts before enterprise deployment freezes block production releases.

How it works

A recurring schedule trigger wakes the flow every 15 minutes to initiate Spinnaker delivery surveillance.
A containerized Python script polls the Spinnaker monitoring daemon Prometheus metrics endpoint.
The script extracts Orca execution task queue depths (orca_queue_depth) and pending message backlogs in Redis.
The script monitors stalled pipeline execution stages and Clouddriver cloud provider HTTP 500 error rates.
A decision gate evaluates whether task queue backlogs or stalled stages breach deployment SLA margins.
When release stalls occur, high-priority notifications dispatch to Release Engineering channels with runbooks.
A structured audit manifest is emitted for enterprise compliance records and CD pipeline telemetry archives.

Architecture diagram

flowchart TD
    A[Schedule Trigger: Every 15 Mins] --> B[scan_spinnaker_queue_metrics]
    B --> C{evaluate_spinnaker_threshold}
    C -- Alert Triggered --> D[notify_slack_platform_team]
    C -- Nominal State --> E[log_nominal_spinnaker_state]
    D --> F[emit_audit_manifest]
    E --> F[emit_audit_manifest]

Architecture components

Trigger (periodic_spinnaker_audit): Recurring cron schedule executing every 15 minutes for proactive CD surveillance.
Scraper Task (scan_spinnaker_queue_metrics): Docker-isolated Python script polling Spinnaker metrics with error handling.
Decision Gate (evaluate_spinnaker_threshold): Evaluates Orca queue depth and stalled stages against user thresholds.
Notification Task (notify_slack_platform_team): Sends rich Slack alert with queue counts, stalled stages, and runbooks.
Compliance Manifest (emit_audit_manifest): Emits structured JSON summary for delivery audit archives.

Key capabilities

Continuous surveillance of Spinnaker Orca task orchestration queue depth and worker capacity.
Early warning when cloud provider rate limits stall canary analysis or blue-green deployments.
Detection of Clouddriver Kubernetes API timeouts before deployment pipelines timeout completely.
Standardized Slack alerting with operational runbook instructions for release engineers.

What you get

Real-time visibility into Spinnaker delivery platform throughput and pipeline execution health.
Prevention of silent deployment deadlocks caused by zombie tasks in underlying Redis queues.
Automated Slack incident notifications with execution queue metrics and remediation advice.
Clean compliance manifests recording CD platform reliability for change management audit trails.

Who it's for

Release Engineers maintaining enterprise Spinnaker multi-cloud deployment platforms.
Platform Engineering Teams operating automated progressive delivery and canary pipelines.
Site Reliability Engineers safeguarding zero-downtime microservice deployments across clouds.
DevOps Teams operating high-frequency continuous delivery pipelines on Kubernetes and AWS.

Why orchestrate this with Kestra

Decoupled Architecture: Scrapes metrics externally without requiring root access to Spinnaker microservices.
Deterministic Scheduling: Native cron scheduling ensures consistent surveillance without cron daemon drift.
Isolated Execution: Runs Python scrapers inside isolated Docker containers with pre-installed libraries.
Enterprise Auditability: Generates structured execution histories and JSON manifests for regulatory reporting.

Use cases

Kubernetes Rate Limiting: Catch when Clouddriver exhausts Kubernetes API server client request quotas.
Zombie Execution Purging: Detect pipelines that remain stuck in running state due to disconnected worker threads.
Redis Queue Saturation: Identify when Redis backing Orca exhausts memory and delays task dispatch.
Canary Stage Timeout Surveillance: Monitor Kayenta canary judge stages waiting on telemetry providers.

Inputs

Name	Type	Default	Description
`spinnaker_metrics_url`	STRING	`http://spinnaker-monitoring.spinnaker.svc.cluster.local:8008`	Base HTTP URL exposing Spinnaker Prometheus metrics.
`target_spinnaker_application`	STRING	`core-payment-gateway`	Target Spinnaker application identifier under active surveillance.
`max_backlog_tasks_threshold`	INT	`50`	Maximum allowable Orca backlog tasks before alerting.
`lookback_minutes`	INT	`15`	Lookback evaluation window in minutes for pipeline telemetry.
`alert_on_spinnaker_failures`	BOOL	`true`	Toggle Slack alert dispatch upon detecting execution stalls or backlogs.

Expected outputs

{{ outputs.scan_spinnaker_queue_metrics.vars.orca_queue_depth }}: Current depth of pending tasks in Orca execution queue.
{{ outputs.scan_spinnaker_queue_metrics.vars.stalled_stages_count }}: Count of pipeline stages stalled beyond execution threshold.
{{ outputs.scan_spinnaker_queue_metrics.vars.clouddriver_errors_count }}: Cumulative count of Clouddriver cloud provider errors.
{{ outputs.scan_spinnaker_queue_metrics.vars.active_pipelines_count }}: Total number of pipelines actively executing across applications.
{{ outputs.scan_spinnaker_queue_metrics.vars.is_alerting }}: Boolean flag indicating whether failure thresholds were breached.
{{ outputs.emit_audit_manifest.value }}: Structured JSON audit manifest emitted for compliance archives.

Prerequisites

Spinnaker cluster deployed with spinnaker-monitoring daemon active and Prometheus port 8008 open.
Metrics port reachable from the Kestra worker execution environment.
Docker daemon running on the Kestra host to execute containerized Python tasks.

Secrets

SLACK_WEBHOOK_URL: Incoming webhook URL for the target Release Engineering Slack alerting channel.

Quick start

Ensure Spinnaker monitoring daemon exposes Prometheus metrics on port 8008.
Configure the SLACK_WEBHOOK_URL secret in your Kestra namespace settings.
Specify your Spinnaker application name in target_spinnaker_application.
Execute a manual test run via the Kestra UI to verify metric ingestion.
Enable the periodic_spinnaker_audit trigger to establish automated continuous surveillance.

Common pitfalls and troubleshooting

Redis Queue Eviction: If Redis runs out of memory, task state is dropped and pipelines hang indefinitely.
Cloud Provider API Throttling: AWS or GCP throttling causes Clouddriver caching agent threads to back up.
Orca Threadpool Exhaustion: High concurrent pipeline triggers saturate Orca worker threadpools; scale Orca pods.
Echo Event Dispatch Latency: Delays in Echo event bus prevent pipeline trigger notifications reaching Orca.

How to extend

Automated Zombie Pipeline Cancellation: Trigger a REST API call to cancel pipelines running longer than 4 hours.
Orca Horizontal Autoscaling: Chain a task to scale Orca deployment replicas when queue depth exceeds 100.
PagerDuty Escalation: Escalate to on-call release engineers if stalled stages exceed 10 across the cluster.

Spinnaker Pipeline Execution Stall and Orca Backlog Sentinel

How it works

Architecture diagram

Architecture components

Key capabilities

What you get

Who it's for

Why orchestrate this with Kestra

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Spinnaker Pipeline Execution Stall and Orca Backlog Sentinel

How it works

Architecture diagram

Architecture components

Key capabilities

What you get

Who it's for

Why orchestrate this with Kestra

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Cloud Blueprints

Monitor FastAPI Synthetic Latency SLOs and Alert SRE Teams via Slack

Fastly Surrogate Key Purge for CMS Content Updates

Fastly Edge Error Spike Alert with the Stats Trigger

Fastly Month-to-Date Cost Guard

Fluent Bit Output Buffer Retry and Chunk Drop Sentinel

Falco Cloud Native Runtime Threat and Security Priority Event Sentinel

More Infrastructure Blueprints

Fastly Emergency Purge All with Confirmation Gate

Set up alerts for failed workflow executions using Discord

Faktory Job Enqueue Path Reachability Sentinel

Purge Fastly Cache on Deploy via Webhook

Fastly Surrogate Key Purge for CMS Content Updates

Fastly Month-to-Date Cost Guard

New to Kestra?