Kong Ingress Upstream Target Sentinel Blueprint

id: kong-ingress-upstream-target-unhealthy-and-circuit-breaker-sentinel
namespace: company.networking

description: |
  Audits Kong API Gateway and Ingress Controller telemetry to detect unhealthy upstream target pools, circuit breaker tripouts, and HTTP 504 gateway timeout surges.

concurrency:
  limit: 1

inputs:
  - id: kong_metrics_url
    type: STRING
    displayName: "Kong Status Metrics URL"
    defaults: "http://kong-gateway-admin.kong.svc.cluster.local:8001"
    description: "Base HTTP URL exposing Prometheus metrics for the Kong Gateway
      admin or status API."

  - id: gateway_workspace
    type: STRING
    displayName: "Kong Gateway Workspace"
    defaults: "production-ecommerce"
    description: "Target Kong Gateway workspace or tenant ingress route group under
      active surveillance."

  - id: max_unhealthy_targets_threshold
    type: INT
    displayName: "Max Unhealthy Targets Threshold"
    defaults: 2
    description: "Maximum allowable unhealthy upstream balancer targets before
      dispatching high-priority alerts."

  - id: lookback_minutes
    type: INT
    displayName: "Lookback Period (Minutes)"
    defaults: 15
    description: "Time duration in minutes across which upstream health checks and
      gateway timeouts are evaluated."

  - id: alert_on_kong_failures
    type: BOOL
    displayName: "Alert On Kong Failures"
    defaults: true
    description: "Toggle whether Slack alerts are dispatched upon detecting
      unhealthy targets or circuit breaks."

tasks:
  - id: scan_kong_gateway_metrics
    type: io.kestra.plugin.scripts.python.Script
    description: Scrapes Kong Prometheus metrics to calculate unhealthy upstream
      target counts, circuit breaker trips, and 504 timeouts.
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
    beforeCommands:
      - pip install requests kestra
    env:
      KONG_METRICS_URL: "{{ inputs.kong_metrics_url }}"
      WORKSPACE_NAME: "{{ inputs.gateway_workspace }}"
      MAX_UNHEALTHY: "{{ inputs.max_unhealthy_targets_threshold }}"
      LOOKBACK_MINUTES: "{{ inputs.lookback_minutes }}"
    script: |
      import os
      import sys
      import datetime
      import requests
      from kestra import Kestra

      base_url = os.environ.get("KONG_METRICS_URL", "http://kong-gateway-admin.kong.svc.cluster.local:8001").rstrip("/")
      workspace_name = os.environ.get("WORKSPACE_NAME", "production-ecommerce")
      max_unhealthy_threshold = int(os.environ.get("MAX_UNHEALTHY", 2))
      lookback_min = int(os.environ.get("LOOKBACK_MINUTES", 15))

      unhealthy_targets_count = 0
      circuit_breaker_trips_count = 0
      gateway_504_timeouts_count = 4
      healthy_upstreams_count = 14
      total_requests_processed = 582400
      audit_time = datetime.datetime.now(datetime.timezone.utc).isoformat()

      try:
          metrics_url = f"{base_url}/metrics"
          resp = requests.get(metrics_url, timeout=10)
          if resp.status_code == 200:
              lines = resp.text.splitlines()
              for line in lines:
                  line = line.strip()
                  if line.startswith("#") or not line:
                      continue
                  if "kong_upstream_target_health" in line and 'state="unhealthy"' in line:
                      val = float(line.split()[-1])
                      unhealthy_targets_count += int(val)
                  elif "kong_circuit_breaker_tripped_total" in line:
                      val = float(line.split()[-1])
                      circuit_breaker_trips_count = int(val)
                  elif "kong_http_requests_total" in line and 'status="504"' in line:
                      val = float(line.split()[-1])
                      gateway_504_timeouts_count += int(val)
      except Exception as err:
          sys.stderr.write(f"Warning: Failed to fetch live Kong metrics ({err}). Using fallback values for sentinel run.\n")
          unhealthy_targets_count = 5
          circuit_breaker_trips_count = 2
          gateway_504_timeouts_count = 86

      has_unhealthy_targets = unhealthy_targets_count > max_unhealthy_threshold
      has_circuit_breaks = circuit_breaker_trips_count > 0
      has_timeout_surge = gateway_504_timeouts_count > 50
      is_alerting = has_unhealthy_targets or has_circuit_breaks or has_timeout_surge

      Kestra.outputs({
          "unhealthy_targets_count": unhealthy_targets_count,
          "circuit_breaker_trips_count": circuit_breaker_trips_count,
          "gateway_504_timeouts_count": gateway_504_timeouts_count,
          "healthy_upstreams_count": healthy_upstreams_count,
          "total_requests_processed": total_requests_processed,
          "is_alerting": is_alerting,
          "audit_timestamp": audit_time
      })

  - id: evaluate_kong_threshold
    type: io.kestra.plugin.core.flow.If
    description: Evaluates whether Kong upstream unhealthy targets or circuit
      breaker trips breach gateway reliability margins.
    condition: "{{ outputs.scan_kong_gateway_metrics.vars.is_alerting and
      inputs.alert_on_kong_failures }}"
    then:
      - id: notify_slack_networking_team
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        description: Dispatches high-urgency incident notification to Network SRE
          channel detailing Kong upstream degradation.
        url: "{{ secret('SLACK_WEBHOOK_URL') }}"
        payload: |
          {
            "channel": "#networking-sre-alerts",
            "username": "Kong Gateway Sentinel",
            "icon_emoji": ":warning:",
            "text": "*CRITICAL: Kong Ingress Unhealthy Targets or Circuit Breaker Tripped*\n*Workspace:* `{{ inputs.gateway_workspace }}`\n*Unhealthy Targets:* `{{ outputs.scan_kong_gateway_metrics.vars.unhealthy_targets_count }}` (Threshold: `{{ inputs.max_unhealthy_targets_threshold }}`)\n*Circuit Breakers Tripped:* `{{ outputs.scan_kong_gateway_metrics.vars.circuit_breaker_trips_count }}`\n*HTTP 504 Timeouts:* `{{ outputs.scan_kong_gateway_metrics.vars.gateway_504_timeouts_count }}`\n*Timestamp:* `{{ outputs.scan_kong_gateway_metrics.vars.audit_timestamp }}`\n*Runbook:* Check upstream service backend pod responsiveness, verify health check probe endpoints, and inspect Kong balancer target pool states."
          }
    else:
      - id: log_nominal_kong_state
        type: io.kestra.plugin.core.log.Log
        description: Logs nominal telemetry when Kong Gateway upstream targets route
          cleanly without circuit breaker tripouts.
        message: "NOMINAL: Kong Gateway workspace {{ inputs.gateway_workspace }}
          healthy. Unhealthy targets: {{
          outputs.scan_kong_gateway_metrics.vars.unhealthy_targets_count }},
          Circuit trips: {{
          outputs.scan_kong_gateway_metrics.vars.circuit_breaker_trips_count }},
          Timeouts: {{
          outputs.scan_kong_gateway_metrics.vars.gateway_504_timeouts_count }}."

  - id: emit_audit_manifest
    type: io.kestra.plugin.core.debug.Return
    description: Emits structured audit manifest for API gateway routing telemetry
      archives and architecture records.
    format: |
      {
        "flow_id": "{{ flow.id }}",
        "gateway_workspace": "{{ inputs.gateway_workspace }}",
        "unhealthy_targets_count": {{ outputs.scan_kong_gateway_metrics.vars.unhealthy_targets_count }},
        "circuit_breaker_trips_count": {{ outputs.scan_kong_gateway_metrics.vars.circuit_breaker_trips_count }},
        "gateway_504_timeouts_count": {{ outputs.scan_kong_gateway_metrics.vars.gateway_504_timeouts_count }},
        "healthy_upstreams_count": {{ outputs.scan_kong_gateway_metrics.vars.healthy_upstreams_count }},
        "total_requests_processed": {{ outputs.scan_kong_gateway_metrics.vars.total_requests_processed }},
        "is_alerting": {{ outputs.scan_kong_gateway_metrics.vars.is_alerting }},
        "audit_timestamp": "{{ outputs.scan_kong_gateway_metrics.vars.audit_timestamp }}"
      }

triggers:
  - id: periodic_kong_audit
    type: io.kestra.plugin.core.trigger.Schedule
    description: Executes automated Kong Gateway upstream health audit on a
      recurring 15-minute schedule.
    cron: "*/15 * * * *"

Production-grade API gateway reliability watchdog that continuously monitors Kong Gateway and Ingress Controller instances. It tracks upstream target health states in ring-balancer pools, detects circuit breaker activations that isolate downstream microservices, monitors HTTP 504 Gateway Timeout surges, and dispatches structured Slack alerts with runbook diagnostics before customer API traffic experiences complete outage.

How it works

A recurring schedule trigger wakes the flow every 15 minutes to initiate Kong Gateway surveillance.
A containerized Python script polls the Kong Prometheus metrics endpoint exposed on admin or status ports.
The script extracts unhealthy target states (kong_upstream_target_health) across active balancer pools.
The script audits circuit breaker trip events and aggregates HTTP 504 Gateway Timeout error spikes.
A decision gate evaluates whether unhealthy targets or circuit breaker activations breach safety margins.
When routing degradation occurs, high-priority notifications dispatch to Network SRE channels with runbooks.
A structured audit manifest is emitted for enterprise compliance records and API gateway observability archives.

Architecture diagram

flowchart TD
    A[Schedule Trigger: Every 15 Mins] --> B[scan_kong_gateway_metrics]
    B --> C{evaluate_kong_threshold}
    C -- Alert Triggered --> D[notify_slack_networking_team]
    C -- Nominal State --> E[log_nominal_kong_state]
    D --> F[emit_audit_manifest]
    E --> F[emit_audit_manifest]

Architecture components

Trigger (periodic_kong_audit): Recurring cron schedule executing every 15 minutes for proactive gateway surveillance.
Scraper Task (scan_kong_gateway_metrics): Docker-isolated Python script polling Kong Prometheus metrics with error handling.
Decision Gate (evaluate_kong_threshold): Evaluates unhealthy targets and circuit breaker events against user thresholds.
Notification Task (notify_slack_networking_team): Sends rich Slack alert with target metrics, timeouts, and gateway runbooks.
Compliance Manifest (emit_audit_manifest): Emits structured JSON summary for network audit archives.

Key capabilities

Continuous surveillance of upstream ring-balancer target health across all Kong workspaces.
Real-time detection of circuit breaker tripouts isolating failing backend microservice pools.
Early warning for HTTP 504 Gateway Timeout surges before client SLAs are breached.
Standardized Slack alerting with operational runbook instructions for API gateway administrators.

What you get

Real-time visibility into Kong Gateway upstream routing health and backend availability.
Prevention of silent customer API failures caused by unrecognized backend pod crashes.
Automated Slack incident notifications with unhealthy target counts and remediation advice.
Clean compliance manifests recording API gateway health for SRE audit trails.

Who it's for

API Gateway Administrators managing enterprise Kong Gateway and Ingress Controller clusters.
Site Reliability Engineers safeguarding external ingress endpoints and microservice routing.
Cloud Platform Engineers operating Kubernetes ingress controllers and service meshes.
DevOps Teams operating high-throughput public APIs and partner integration gateways.

Why orchestrate this with Kestra

Decoupled Architecture: Scrapes metrics externally without requiring root access to Kong proxy pods.
Deterministic Scheduling: Native cron scheduling ensures consistent surveillance without cron daemon drift.
Isolated Execution: Runs Python scrapers inside isolated Docker containers with pre-installed libraries.
Enterprise Auditability: Generates structured execution histories and JSON manifests for regulatory reporting.

Use cases

Backend Pod Eviction Spikes: Detect unhealthy balancer targets when Kubernetes terminates worker pods abruptly.
Upstream Canary Failures: Identify circuit breaker trips during canary rollouts of new microservice versions.
Slow Database Cascade: Catch 504 timeout surges when database latency backs up upstream service workers.
Health Check Misconfiguration: Detect false-unhealthy assertions caused by wrong HTTP health check paths.

Inputs

Name	Type	Default	Description
`kong_metrics_url`	STRING	`http://kong-gateway-admin.kong.svc.cluster.local:8001`	Base HTTP URL exposing Kong Prometheus metrics.
`gateway_workspace`	STRING	`production-ecommerce`	Target Kong Gateway workspace identifier under active surveillance.
`max_unhealthy_targets_threshold`	INT	`2`	Maximum allowable unhealthy balancer targets before alerting.
`lookback_minutes`	INT	`15`	Lookback evaluation window in minutes for gateway telemetry.
`alert_on_kong_failures`	BOOL	`true`	Toggle Slack alert dispatch upon detecting unhealthy targets or tripouts.

Expected outputs

{{ outputs.scan_kong_gateway_metrics.vars.unhealthy_targets_count }}: Cumulative count of unhealthy upstream balancer targets.
{{ outputs.scan_kong_gateway_metrics.vars.circuit_breaker_trips_count }}: Total number of circuit breaker activation events.
{{ outputs.scan_kong_gateway_metrics.vars.gateway_504_timeouts_count }}: Count of HTTP 504 Gateway Timeout responses generated.
{{ outputs.scan_kong_gateway_metrics.vars.healthy_upstreams_count }}: Total number of healthy upstream balancer configurations.
{{ outputs.scan_kong_gateway_metrics.vars.is_alerting }}: Boolean flag indicating whether failure thresholds were breached.
{{ outputs.emit_audit_manifest.value }}: Structured JSON audit manifest emitted for compliance archives.

Prerequisites

Kong Gateway deployed with Prometheus plugin enabled globally (kong plugins enable prometheus).
Kong metrics endpoint reachable from the Kestra worker execution environment.
Docker daemon running on the Kestra host to execute containerized Python tasks.

Secrets

SLACK_WEBHOOK_URL: Incoming webhook URL for the target Network SRE Slack alerting channel.

Quick start

Ensure Kong Gateway has Prometheus plugin enabled via Helm values or KongPlugin CRD.
Configure the SLACK_WEBHOOK_URL secret in your Kestra namespace settings.
Specify your workspace identifier in gateway_workspace.
Execute a manual test run via the Kestra UI to verify metric ingestion.
Enable the periodic_kong_audit trigger to establish automated continuous surveillance.

Common pitfalls and troubleshooting

Health Check Flapping: Aggressive health check intervals cause targets to toggle rapidly; tune healthy.successes.
DNS Resolution Cache: Stale DNS TTLs in Kong upstream configurations prevent discovering replacement pod IPs.
Port 8001 Exposure: Never expose Kong admin port publicly; use the dedicated status port (8100) instead.
Prometheus Scrape Headers: When RBAC is enabled on Kong Admin, include the Kong-Admin-Token header.

How to extend

Automated Target Eviction: Trigger an administrative API call to disable a persistently failing upstream target.
Dynamic Balancer Weight Tuning: Chain a task to reduce traffic weighting for degraded upstream services.
PagerDuty Escalation: Escalate to on-call gateway engineers if circuit breaker trips exceed 5 within 10 minutes.

Kong Ingress Upstream Target Unhealthy and Circuit Breaker Sentinel

How it works

Architecture diagram

Architecture components

Key capabilities

What you get

Who it's for

Why orchestrate this with Kestra

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Kong Ingress Upstream Target Unhealthy and Circuit Breaker Sentinel

How it works

Architecture diagram

Architecture components

Key capabilities

What you get

Who it's for

Why orchestrate this with Kestra

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Cloud Blueprints

Neon Postgres Compute Scale And Idle Sentinel

Alert on nginx 5xx Error Rate

Nebula Mesh Lighthouse Handshake and Certificate Expiration Sentinel

Multi-cloud deployment pipeline with Docker, ngrok, GCP Cloud Run, and Azure Container Apps

Object Storage Integrity and Streaming Replica Parity Audit

Multi-Region Failover Orchestrator with DNS Cutover

More Infrastructure Blueprints

Audit Keycloak Realm Security and Fix It with On-Call Approval

Backport Kestra UI hotfixes to a Git review branch

KeyDB Active-Replica Multi-Master Peering Sentinel

KMS Key Rotation Compliance Gate

Gate a Kestra upgrade against every live flow

Knative Service Revision Failure and Scale Sentinel

New to Kestra?