Schedule icon
Request icon
Switch icon
Log icon
SlackIncomingWebhook icon

SearXNG Metasearch Engine Health and Upstream Rate-Limit Sentinel

Audit SearXNG metasearch instance health, detect upstream provider rate-limits and suspended engines, and dispatch automated Slack incident alerts.

Categories
AI

Automated health gate and availability sentinel that monitors SearXNG privacy-preserving metasearch instances against upstream engine rate-limiting, CAPTCHA blocks, and engine suspensions. Dispatches structured Slack incident notifications to prevent silent degradation of AI search agents and RAG retrieval pipelines.

How it works

  1. A recurrent scheduled trigger initiates the metasearch audit every 4 hours, or triggers on-demand via webhook or manual dispatch before downstream AI runs.
  2. The probe_instance_health task queries GET /healthz on the target SearXNG instance to confirm HTTP reachability and base process readiness.
  3. The probe_synthetic_search task runs a live synthetic search query against GET /search?q=...&format=json to evaluate end-to-end response parsing, latency, and provider aggregation.
  4. Pebble expressions inspect the returned JSON payload, evaluating number_of_results against min_expected_results and cross-referencing unresponsive_engines against the monitored_engines allowlist.
  5. The route_engine_health_status flow-control Switch evaluates the classified operational state:
    • COMPLIANT: Logs clean audit metrics when all monitored providers respond successfully and return sufficient results.
    • DEGRADED: Formats and dispatches a rich Slack warning detailing which upstream engines are unresponsive or suspended.
    • SEARCH_EMPTY: Dispatches a critical Slack outage alert notifying AI platform engineers that zero results were aggregated, indicating universal bot blocking or IP blacklisting.
  6. A final log_audit_summary task records execution audit telemetry, while an errors block intercepts connection timeouts or gateway failures with automated Slack escalations.

What you get

  • Early warning detection for upstream search engine rate-limiting (HTTP 429) and bot-detection IP blocks.
  • Zero-downtime visibility into SearXNG engine suspensions before downstream RAG agents fail.
  • Automated Slack incident routing with detailed diagnostic attachments listing specific failing providers.
  • Continuous synthetic search verification validating real-time response payload formatting.
  • Centralized audit trail of metasearch reliability metrics preserved in Kestra's execution history.

Who it's for

  • AI Engineers and Data Scientists building agentic search and retrieval-augmented generation (RAG) pipelines.
  • Platform Engineers and SREs hosting self-managed SearXNG metasearch clusters.
  • DevOps Teams operating privacy-focused search infrastructure and web scraping gateways.

Why orchestrate this with Kestra

  • Declarative YAML Configuration: Maintain search engine health policies, monitored provider lists, and alert thresholds in version-controlled workflows.
  • Native Secret Isolation: Securely store Slack Incoming Webhook endpoints in namespace secret stores using {{ secret(...) }} without plaintext exposure.
  • Proactive Failure Gates: Integrate this sentinel as an upstream dependency before running scheduled RAG ingestion batches to prevent incomplete data gathering.
  • Resilient Flow-Control: Combine Kestra's native Switch, Log, and errors tasks to distinguish between local container failures and upstream provider rate-limits.

Prerequisites

  • A running SearXNG instance (accessible over HTTP/HTTPS) with JSON output format enabled (search.formats: [html, json]).
  • Network connectivity from the Kestra worker or runner environment to the SearXNG server.
  • A Slack Incoming Webhook URL configured with permissions to post into your target operations channel.

Secrets

  • SLACK_WEBHOOK_URL: Slack Incoming Webhook endpoint URL used for operational alerts.

Quick start

  1. Configure the SLACK_WEBHOOK_URL secret in your Kestra namespace settings.
  2. Import this blueprint into your Kestra instance under your desired namespace (e.g., company.ai).
  3. Click Execute with default inputs or provide your custom searxng_host to validate connectivity against your instance.
  4. Review execution logs under the Executions tab to confirm that synthetic search results and engine metrics are parsed correctly.
  5. Enable the scheduled_sentinel_audit trigger to establish recurrent 4-hour automated metasearch health evaluations.

How to extend

  • Automated Engine Toggling: Extend the degraded branch with an HTTP request to reload SearXNG configuration or rotate outbound proxy IPs when rate limits occur.
  • Proxy Pool Health Check: Add parallel probe tasks to verify upstream Tor, SOCKS5, or residential proxy gateway reachability.
  • PagerDuty Escalation: Route critical zero-result outages directly to on-call engineers via PagerDuty or Opsgenie.
  • Multi-Query Canary Suite: Expand the synthetic validation into a Kestra Loop task testing varied search categories (e.g., general, science, news, IT).

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.