Detect Qdrant embedding drift and verify bounded re-embedding
Compare live embeddings with stored vectors and repair an approved batch only when held-out retrieval has degraded.
Investigate database connection spikes and lock contention with DuckDB diagnostics and Kestra AI agent to synthesize copy-paste remediation commands.
Production database incidents (connection pool exhaustion, lock tree contention, hung transaction spikes) frequently trigger high-severity alerts outside standard business hours. When an incident occurs, on-call Site Reliability Engineers (SREs) are awakened and lose critical minutes manually running routine diagnostic queries (pg_stat_activity, lock dependency trees, active transactions) to identify the offending session.
This blueprint implements an autonomous Tier-1 AI SRE investigation agent. When a connection pool or latency alert fires, the workflow ingests process session telemetry and runs relational lock tree diagnostics using in-memory DuckDB. Next, Kestra's AI plugin evaluates the lock graph against production database failure archetypes with strict JSON Schema constraints. It isolates the exact root-cause blocker PID and offending query, checks whether downstream checkout queries are hung, and delivers a copy-paste remediation SQL command (pg_terminate_backend) with architectural prevention guidance directly to Slack.
collect_diagnostic_telemetry (io.kestra.plugin.jdbc.duckdb.Queries): Ingests the active database session snapshot and computes lock dependency trees, isolating the longest-running unblocked root session.investigate_incident (io.kestra.plugin.ai.agent.AIAgent): Evaluates session telemetry and alert symptoms using Google Gemini (gemini-2.5-flash) with structured JSON Schema enforcement (CRITICAL_LOCK_CONTENTION, LONG_RUNNING_IDLE_TRANSACTION, HIGH_VOLUME_TRAFFIC_SPIKE), isolating the culprit PID and drafting the remediation SQL command.evaluate_severity_gate (io.kestra.plugin.core.flow.If):CRITICAL: Posts an actionable investigation summary card to #sre-incidents in Slack containing the culprit PID, offending SQL, and copy-paste kill command.alert_investigation_failure (errors block): Catches downstream connectivity failures or unhandled exceptions and notifies the on-call channel.pluginDefaults keeping Slack webhook URLs out of individual task declarations.Scripting database incident triage with ad-hoc cron jobs or standalone Python scripts creates brittle infrastructure that lacks centralized secret management, observability, and structured AI response guarantees. Orchestrating with Kestra provides declarative scheduling, authenticated webhooks for APM alarms (Datadog, Prometheus), in-memory DuckDB analytical parsing, structured JSON Schema model reasoning, and an immutable execution audit trail in a single version-controlled YAML workflow.
pg_terminate_backend), ensure the target PID is a client worker session rather than an internal engine worker (such as the autovacuum launcher or logical replication worker), which can trigger crash recovery.ALTER TABLE ... ADD COLUMN) hold an AccessExclusiveLock, blocking all read and write traffic. Production migrations should enforce strict session lock timeouts (SET lock_timeout = '3s';).gemini-2.5-flash.GEMINI_API_KEY: API key for Gemini foundational model execution.SLACK_WEBHOOK_URL: Slack webhook endpoint for alert delivery.WEBHOOK_KEY: Authentication secret for event-driven webhook execution.GEMINI_API_KEY and SLACK_WEBHOOK_URL to your Kestra namespace secrets.alert_webhook_trigger.This blueprint has been created by sheikhlimon