Schedule icon
Request icon
Switch icon
Log icon
SlackIncomingWebhook icon

Vespa Cluster Convergence and Node Health Sentinel

Audit Vespa vector search cluster health, detect config generation convergence drift, and alert Slack on node degradation or heap exhaustion.

Categories
AI

Automated health gate and convergence sentinel that monitors Vespa online big data serving and vector search clusters for container node readiness, config generation synchronization, and JVM heap memory saturation. Dispatches structured Slack incident notifications to prevent search latency spikes and schema drift in real-time recommendation and retrieval pipelines.

How it works

  1. A recurrent scheduled trigger initiates the cluster audit every 3 hours, or executes on-demand via webhook or manual dispatch before deploying schema updates.
  2. The probe_node_health task queries GET /state/v1/health on the target Vespa container node to confirm HTTP service status and process readiness.
  3. The probe_application_status task queries GET /ApplicationStatus to inspect active application package metadata, deployed generation numbers, and cluster model definitions.
  4. The probe_cluster_metrics task queries GET /state/v1/metrics to capture container JVM memory allocation, free heap counters, and indexing throughput telemetry.
  5. Pebble expressions inspect the composite response payloads, evaluating health status code, comparing active generation against expected_application_generation, and verifying free heap memory against min_free_heap_bytes.
  6. The evaluate_cluster_health flow-control Switch evaluates the classified operational posture:
    • COMPLIANT: Logs clean operational metrics when the container is responsive, generation matches expectation, and heap headroom is adequate.
    • DEGRADED_HEAP: Formats and dispatches a rich Slack warning notifying engineers that JVM heap headroom has fallen below safety margins.
    • GENERATION_DESYNC: Dispatches a high-priority Slack alert detailing a configuration generation mismatch, identifying unapplied schemas or stalled cluster convergence.
    • NODE_DOWN: Posts an immediate critical incident alert notifying on-call engineers of service failure or unhealthy node states.
  7. A final log_audit_summary task records execution audit telemetry, and an errors block intercepts connection timeouts or network unreachability with automated Slack escalations.

What you get

  • Early detection of stalled application package deployments and schema desynchronizations across container nodes.
  • Continuous JVM heap saturation surveillance preventing container out-of-memory (OOM) crashes and query latency spikes.
  • Automated Slack incident routing with actionable diagnostic payload attachments including generation IDs and status descriptions.
  • Safe operational gate validating node convergence before routing production traffic or commencing batch embedding feeds.
  • Centralized audit trail of search engine health records preserved within Kestra's execution history.

Who it's for

  • Platform Engineers and SREs maintaining production Vespa vector search and recommendation clusters.
  • AI Engineers and Search Architects deploying neural ranking models and hybrid vector/lexical retrieval pipelines.
  • Data Infrastructure Teams managing real-time document feeds and high-throughput analytical serving systems.

Why orchestrate this with Kestra

  • Declarative YAML Operations: Codify cluster health thresholds, generation expectations, and alert escalation pathways in version-controlled workflows.
  • Native Secret Isolation: Securely store Slack Incoming Webhook URLs and internal endpoint credentials in namespace secret stores via {{ secret(...) }}.
  • Resilient Multi-Probe Architecture: Chain sequential HTTP probes and evaluate multi-dimensional status indicators using native Pebble expressions and Switch branches.
  • Proactive Deployment Safeguards: Integrate this flow as an automated post-deployment validation step to confirm cluster convergence after rolling application updates.

Prerequisites

  • A running Vespa container or cluster endpoint (accessible over HTTP/HTTPS) with standard state and metrics APIs exposed.
  • Network connectivity from the Kestra worker or runner environment to the Vespa server.
  • A Slack Incoming Webhook URL configured with permissions to post into your target operations channel.

Secrets

  • SLACK_WEBHOOK_URL: Slack Incoming Webhook endpoint URL used for operational alerts.

Quick start

  1. Configure the SLACK_WEBHOOK_URL secret in your Kestra namespace settings.
  2. Import this blueprint into your Kestra instance under your desired namespace (e.g., company.ai).
  3. Click Execute with default inputs or provide your custom vespa_host to validate connectivity against your instance.
  4. Review execution logs under the Executions tab to confirm that health codes, generation numbers, and heap metrics are extracted accurately.
  5. Enable the scheduled_sentinel_audit trigger to establish recurrent 3-hour automated cluster health evaluations.

How to extend

  • Automated Config Activation: Extend the desync branch with an HTTP request to the Vespa config server (POST /application/v2/tenant/.../prepareandactivate) to re-trigger convergence.
  • Proton Document Storage Probe: Add an additional probe task querying content node /state/v1/metrics to monitor disk resource limits and attribute memory usage.
  • PagerDuty Escalation: Route critical NODE_DOWN states directly to PagerDuty or Opsgenie on-call schedules.
  • Multi-Node Cluster Sweep: Combine with a Kestra Loop task to iterate across an array of distributed container node URLs.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.