AI Incident Triage with RAG and Slack Notifications

Turn raw incident payloads into grounded, actionable triage guidance. This Kestra blueprint receives an incident over a webhook, embeds your operations knowledge base into the built-in Kestra KV vector store, retrieves the most relevant playbook sections with RAG, and asks Google Gemini for triage steps that always include risk, steps, and verification. The proposal is logged for the audit trail and posted to Slack for the on-call engineer. Because the model is grounded in your own playbooks, it answers with your procedures instead of generic advice, and it returns 'N/A' for tickets outside the knowledge base rather than hallucinating.

How it works

1. The incident_trigger (io.kestra.plugin.core.trigger.Webhook) accepts a POST from monitoring or ITSM tooling, secured by INCIDENT_WEBHOOK_KEY. A JSON incident input provides a fallback payload so the flow also runs manually.
2. The log_incident task (io.kestra.plugin.core.log.Log) records the raw payload.
3. The ingest_kb task (io.kestra.plugin.ai.rag.IngestDocument) embeds an inline knowledge base stub with the gemini-embedding-001 model into io.kestra.plugin.ai.embeddings.KestraKVStore, dropping the previous index each run so the store always mirrors the current playbook.
4. The enrich_with_rag task (io.kestra.plugin.ai.rag.ChatCompletion) retrieves relevant KB segments and asks gemini-3.5-flash-lite for a concise summary, work notes, and a proposed resolution, constrained by a system message that enforces Risk, Steps, and Verification sections.
5. log_incident_response persists the answer in execution logs and post_to_slack (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) delivers it to the on-call channel.

What you get

• An event-driven triage assistant grounded in your own runbooks, not general model knowledge.
• A built-in guardrail: incidents that do not match the KB produce 'N/A' instead of invented steps.
• A consistent triage format (risk, steps, verification) that reads like a runbook entry.
• Full execution history of every incident and every AI proposal for post-incident review.
• Zero external vector database: embeddings live in the Kestra KV store.

Who it's for

• SRE and platform teams that want first-pass triage in Slack seconds after an alert fires.
• IT operations teams standardizing tribal knowledge into playbook-driven responses.
• Teams evaluating RAG for ops who want a working pipeline without standing up vector infrastructure.

Why orchestrate this with Kestra

A standalone script calling an LLM has no trigger surface, no retry semantics, and no audit trail. Kestra provides the webhook entry point, versioned declarative YAML for the whole RAG pipeline, per-task logs that show exactly what the model was asked and answered, and secret management for API keys. The IngestDocument and ChatCompletion tasks from the io.kestra.plugin.ai plugin make embedding and retrieval first-class workflow steps, so re-indexing the KB is just another task run you can schedule, observe, and replay.

Prerequisites

• A Google AI Studio API key with access to Gemini chat and embedding models.
• A Slack incoming webhook URL for the on-call channel.
• A monitoring or ITSM tool able to POST incident JSON to the webhook.

Secrets

• GEMINI_API_KEY: Google Gemini API key used by both the embedding and chat providers.
• SLACK_WEBHOOK: Slack incoming webhook URL for triage notifications.
• INCIDENT_WEBHOOK_KEY: shared secret guarding the Webhook trigger.

Quick start

1. Add the secrets above to your Kestra namespace.
2. Replace the inline knowledge base stub in ingest_kb with your real playbooks, or point fromDocuments at namespace files.
3. Deploy the flow and run it manually: the default incident input simulates an Nginx 502 alert.
4. Wire your monitoring tool to POST incidents to the webhook URL with the configured key.
5. Watch triage proposals arrive in Slack with risk, steps, and verification sections.

How to extend

• Split KB ingestion into a separate scheduled flow so triage runs skip re-embedding.
• Swap KestraKVStore for a production vector store such as pgvector, Weaviate, or Elasticsearch via the corresponding io.kestra.plugin.ai.embeddings types.
• Add io.kestra.plugin.core.flow.If to page on-call via PagerDuty only when severity is high.
• Chain a Pause approval task and an automated remediation step for low-risk fixes like service restarts.
• Replace Gemini with OpenAI, Anthropic, or Ollama by changing the provider blocks.

Links

• Kestra AI plugin docs: https://kestra.io/plugins/plugin-ai
• Slack plugin docs: https://kestra.io/plugins/plugin-notifications