Kubernetes AI Incident Monitoring with Kestra

id: k8s-ai-incident-monitor
namespace: company.team

tasks:
  - id: parallel
    type: io.kestra.plugin.core.flow.Parallel
    tasks:
      - id: collect_pods
        type: io.kestra.plugin.kubernetes.kubectl.Get
        description: Get pod status
        resourceType: pods
        outputFiles:
          - pods.json

      - id: collect_events
        type: io.kestra.plugin.kubernetes.kubectl.Get
        description: Get recent kubernetes events
        resourceType: events
        outputFiles:
          - events.json

      - id: collect_nodes
        type: io.kestra.plugin.kubernetes.kubectl.Get
        description: Get cluster nodes status
        namespace: ""
        resourceType: nodes
        outputFiles:
          - nodes.json

      - id: collect_deployments
        type: io.kestra.plugin.kubernetes.kubectl.Get
        description: Get deployments status
        resourceType: deployments
        outputFiles:
          - deployments.json

      - id: collect_services
        type: io.kestra.plugin.kubernetes.kubectl.Get
        description: Get services status
        resourceType: services
        outputFiles:
          - services.json

      - id: collect_resource_quotas
        type: io.kestra.plugin.kubernetes.kubectl.Get
        description: Monitor resource quotas and limits
        resourceType: resourcequotas
        outputFiles:
          - resource-quotas.json

      - id: collect_logs
        type: io.kestra.plugin.kubernetes.kubectl.Get
        description: Get pod logs for analysis
        namespace: kube-system
        resourceType: pods
        resourcesNames:
          - kube-apiserver-docker-desktop
        outputFiles:
          - kube-logs.json

  - id: ai_incident_analysis
    type: io.kestra.plugin.ai.completion.ChatCompletion
    description: AI-powered analysis of cluster incidents
    provider:
      type: io.kestra.plugin.ai.provider.GoogleGemini
      apiKey: "{{ secret('GOOGLE_API_KEY') }}"
      modelName: gemini-2.5-flash
    messages:
      - type: SYSTEM
        content: |
          You are a Kubernetes expert and site reliability engineer. You'll be given .ion format files uri .Download its content and Analyze the provided cluster data and provide:
          1. A concise summary of the current cluster health
          2. Root cause analysis for any issues found
          3. Impact assessment (severity, affected resources with names)
          4. Specific remediation steps with kubectl commands
          5. Preventive measures to avoid similar issues
          Do not use backticks, quotes, asterisks, or any Markdown formatting.  
          Ensure all text is treated as plain text without any special characters or indentation.  
          If mentioning code or error messages, format them as inline text with no special symbols.  
          Don't use any quotes in your response and if you do, 
          make sure to escape them with back-slashes e.g. \"Command failed with exit code 1\".
      - type: USER
        content: |
          Kubernetes Cluster Analysis Request:
          Pod Status: content from {{ read(outputs.collect_pods.uri) }}
          Recent Events: Content from {{ read(outputs.collect_events.uri) }}
          Node Status: Content from {{ read(outputs.collect_nodes.uri) }}
          Deployment Status: Content from {{ read(outputs.collect_deployments.uri) }}
          Logs: Content from {{ read(outputs.collect_logs.uri) }}
          Please provide a comprehensive analysis and recommendations.

  - id: classify_severity
    type: io.kestra.plugin.scripts.python.Script
    description: Classify incident severity based on AI analysis
    script: |
      import json
      analysis = """{{ outputs.ai_incident_analysis.aiResponse }}"""
      # Extract severity indicators
      severity = "LOW"
      if any(keyword in analysis.lower() for keyword in ['critical', 'down', 'failed', 'error']):
          severity = "HIGH"
      elif any(keyword in analysis.lower() for keyword in ['warning', 'degraded', 'slow']):
          severity = "MEDIUM"
      print(json.dumps({"severity": severity, "analysis": analysis}))

  - id: send_slack_summary
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    runIf: "{{ outputs.classify_severity.vars.severity == 'HIGH' }}"
    description: Send cluster health summary to Slack
    url: https://kestra.io/api/mock # replace with "{{ secret('SLACK_WEBHOOK_URL') }}" for production
    messageText: "Cluster Alert with severity {{
      outputs.classify_severity.vars.severity }}. Full analysis - {{
      outputs.classify_severity.vars.analysis }}"

triggers:
  - id: schedule_monitor
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "*/5 * * * *"

pluginDefaults:
  - type: io.kestra.plugin.kubernetes.kubectl.Get
    forced: false
    values:
      fetchType: STORE
      namespace: default
      connection:
        masterUrl: "{{ secret('K8S_MASTER_URL') }}"
        oauthToken: "{{ secret('K8S_TOKEN') }}"

description: |
  Kubernetes AI-powered incident monitoring and alerting system.
  Monitors cluster health, analyzes issues with AI, and sends intelligent notifications.

Continuously monitor a Kubernetes cluster for health problems and incidents without writing custom controllers or stitching together cron jobs and shell scripts. Every five minutes this flow snapshots the live state of your cluster, hands the raw data to a large language model for site reliability analysis, classifies how serious the situation is, and pages your team on Slack only when something genuinely needs attention. It turns noisy kubectl output into a concise root cause summary with concrete remediation steps, so on-call engineers spend less time triaging and more time fixing.

How it works

A io.kestra.plugin.core.flow.Parallel task fans out seven io.kestra.plugin.kubernetes.kubectl.Get calls at once to collect pods, events, nodes, deployments, services, resourcequotas, and selected kube-system pod logs, each written to its own output file.
io.kestra.plugin.ai.completion.ChatCompletion sends the collected data to Google Gemini (gemini-2.5-flash) acting as an SRE, returning a health summary, root cause analysis, impact assessment, and remediation commands.
A io.kestra.plugin.scripts.python.Script task (classify_severity) scans the AI response for keywords and labels the incident LOW, MEDIUM, or HIGH.
io.kestra.plugin.slack.notifications.SlackIncomingWebhook fires only when runIf evaluates severity as HIGH, posting the full analysis to your channel.
A io.kestra.plugin.core.trigger.Schedule trigger runs the whole pipeline on a */5 * * * * cron.

What you get

Parallel, low-latency collection of the cluster signals that matter most
Plain-language incident reports with named affected resources and kubectl fixes
Severity gating so Slack only lights up for real HIGH events
A hands-off monitor that runs on a schedule with no extra infrastructure

Who it's for

Platform and SRE teams running production Kubernetes
On-call engineers who want triage done before they open a terminal
DevOps teams adding AIOps to existing observability

Why orchestrate this with Kestra

Kubernetes ships a scheduler for pods, not for cross-resource diagnostic workflows. Kestra adds the missing layer: event and schedule triggers, automatic retries on transient API failures, full execution lineage and replay of every snapshot and AI verdict, and a declarative YAML definition you can version in Git. The parallel collection, conditional Slack alerting, and AI step live in one auditable flow instead of scattered scripts.

Prerequisites

A reachable Kubernetes API endpoint and a token with read access to the monitored resources
A Google Gemini API key
A Slack incoming webhook URL for production alerting

Secrets

K8S_MASTER_URL: Kubernetes API master URL
K8S_TOKEN: OAuth token for cluster access
GOOGLE_API_KEY: Google Gemini API key
SLACK_WEBHOOK_URL: Slack incoming webhook (swap in for the mock URL in production)

Quick start

Add the four secrets above to your Kestra instance.
Adjust the namespace in pluginDefaults and the resource selectors to match your cluster.
Replace the mock Slack URL in send_slack_summary with {{ secret('SLACK_WEBHOOK_URL') }}.
Run the flow once manually, then let the five-minute schedule take over.

How to extend

Point it at multiple clusters by parameterizing the connection details via inputs.
Swap Gemini for another provider supported by the AI plugin.
Add severity tiers or route MEDIUM incidents to a different channel or PagerDuty.
Persist each analysis to a database or object store for trend reporting.

Kubernetes AI-powered incident monitoring, analysis, and alerting

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Kubernetes AI-powered incident monitoring, analysis, and alerting

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More Infrastructure Blueprints

Security & Compliance, Detect and Alert on Infrastructure Configuration Drift with Ansible

Ansible with a Dynamic NetBox Inventory and ServiceNow Record

Rolling Ansible Patch Wave with Health Gates

Ansible Scalability, Deploy Configuration Files to Multiple Servers at Scale

Ansible 101, How to Orchestrate and Run a Simple Ansible Playbook

IIS Site Provisioning with SSL Binding and Health Check

More AI Blueprints

Draft GDPR Breach Notifications for Supervisory Authorities and Data Subjects

Generate and Send a Personalized AI-Powered Daily Digest Email Every Morning

Summarize Text with an AI Agent, Generate Multilingual Summaries in Short, Medium, or Long Format

Decide When to Take Action Using AI, Check Real-Time Weather and Trigger Automations

AI Annual Report Finder and Data Strategy Summary

Generate a book chapter summary using Perplexity AI and email it.

New to Kestra?