Quickwit Search Latency & Error Budget Sentinel

id: quickwit-search-latency-and-error-budget-sentinel
namespace: company.team

description: |
  Audit Quickwit search query latency against configured SLA budgets and detect
  error log volume spikes within a rolling lookback window. Evaluates cluster
  health, branches on degraded performance, dispatches Slack notifications,
  and emits audit logs for continuous observability.

inputs:
  - id: quickwit_url
    type: STRING
    displayName: Quickwit REST API URL
    description: Base URL of the Quickwit REST API service, including protocol and
      port. A local Quickwit node listens on port 7280.
    defaults: "http://localhost:7280"

  - id: index_id
    type: STRING
    displayName: Target Index ID
    description: Identifier of the Quickwit index to audit for search latency and
      error log frequency.
    defaults: "app-logs"

  - id: lookback_minutes
    type: INT
    displayName: Error Window Lookback (Minutes)
    description: Duration of the recent time window in minutes to evaluate for error
      document volume spikes.
    defaults: 15

  - id: latency_sla_ms
    type: INT
    displayName: Search Latency SLA Threshold (ms)
    description: Maximum acceptable search query execution duration in milliseconds.
      Queries exceeding this latency trigger an SLA violation alert.
    defaults: 500

  - id: error_budget_count
    type: INT
    displayName: Error Budget Volume Limit
    description: Maximum number of error log documents permitted within the rolling
      lookback window before alerting.
    defaults: 25

  - id: alert_mode
    type: SELECT
    displayName: Sentinel Alert Mode
    description: Operating mode of the sentinel. ALERT dispatches a notification to
      Slack when thresholds breach; AUDIT_ONLY logs findings without paging
      on-call.
    values:
      - "ALERT"
      - "AUDIT_ONLY"
    defaults: "ALERT"

triggers:
  - id: periodic_sentinel_schedule
    type: io.kestra.plugin.core.trigger.Schedule
    description: Runs every 15 minutes to verify search query latency and log error
      rates against operational budgets. Shipped disabled by default for safe
      initial setup.
    cron: "*/15 * * * *"
    disabled: true
    inputs:
      quickwit_url: "http://localhost:7280"
      index_id: "app-logs"
      lookback_minutes: 15
      latency_sla_ms: 500
      error_budget_count: 25
      alert_mode: "ALERT"

tasks:
  - id: probe_index_metadata
    type: io.kestra.plugin.quickwit.index.Get
    description: >
      Retrieve target index metadata from Quickwit, including configuration,
      generation UID, and creation timestamp to confirm index availability.
    url: "{{ inputs.quickwit_url }}"
    index: "{{ inputs.index_id }}"

  - id: probe_error_volume
    type: io.kestra.plugin.quickwit.search.Search
    description: >
      Search for error documents matching severity ERROR or HTTP status 5xx
      within the specified lookback window. Returns total document count.
    url: "{{ inputs.quickwit_url }}"
    index: "{{ inputs.index_id }}"
    query: "severity:ERROR OR status:>=500"
    startTimestamp: "{{ (now() | dateAdd(-inputs.lookback_minutes, 'MINUTES')) | timestamp }}"
    maxHits: 5
    fetchType: FETCH

  - id: probe_canary_latency
    type: io.kestra.plugin.quickwit.search.Search
    description: >
      Execute a lightweight wildcard canary query across the index to measure
      raw search execution latency in microseconds.
    url: "{{ inputs.quickwit_url }}"
    index: "{{ inputs.index_id }}"
    query: "*"
    maxHits: 1
    fetchType: NONE

  - id: evaluate_slo_and_error_budget
    type: io.kestra.plugin.core.flow.If
    description: >
      Evaluate canary search execution duration and recent error document count.
      Triggers the alert branch if search duration exceeds latency_sla_ms or
      error count exceeds error_budget_count.
    condition: "{{ (outputs.probe_canary_latency.elapsedTimeMicros >
      (inputs.latency_sla_ms * 1000)) or (outputs.probe_error_volume.total >
      inputs.error_budget_count) }}"
    then:
      - id: evaluate_alert_mode
        type: io.kestra.plugin.core.flow.If
        description: >
          Check configured alert_mode. ALERT mode dispatches a Slack incoming
          webhook; AUDIT_ONLY mode emits structured logs without paging
          operators.
        condition: "{{ inputs.alert_mode == 'ALERT' }}"
        then:
          - id: send_slack_alert
            type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
            description: >
              Post a structured alert notification to the designated Slack
              channel detailing latency figures, error counts, and execution
              metadata.
            url: "{{ secret('SLACK_WEBHOOK') }}"
            payload: |
              {
                "text": "Quickwit sentinel alert for index {{ inputs.index_id }} on {{ inputs.quickwit_url }}",
                "blocks": [
                  {
                    "type": "header",
                    "text": {
                      "type": "plain_text",
                      "text": "Quickwit Search Latency & Error Budget Alert"
                    }
                  },
                  {
                    "type": "section",
                    "fields": [
                      {
                        "type": "mrkdwn",
                        "text": "*Target Index:*\n{{ inputs.index_id }}"
                      },
                      {
                        "type": "mrkdwn",
                        "text": "*Index UID:*\n{{ outputs.probe_index_metadata.indexUid }}"
                      },
                      {
                        "type": "mrkdwn",
                        "text": "*Canary Latency:*\n{{ (outputs.probe_canary_latency.elapsedTimeMicros / 1000) | int }} ms (SLA: {{ inputs.latency_sla_ms }} ms)"
                      },
                      {
                        "type": "mrkdwn",
                        "text": "*Error Hits ({{ inputs.lookback_minutes }}m):*\n{{ outputs.probe_error_volume.total }} (Budget: {{ inputs.error_budget_count }})"
                      }
                    ]
                  },
                  {
                    "type": "section",
                    "text": {
                      "type": "mrkdwn",
                      "text": "*Execution:* {{ flow.namespace }}.{{ flow.id }} | ID: {{ execution.id }}"
                    }
                  }
                ]
              }
        else:
          - id: log_audit_only_breach
            type: io.kestra.plugin.core.log.Log
            description: >
              Emit structured audit finding when thresholds are breached in
              AUDIT_ONLY mode.
            message: >
              AUDIT_ONLY: Quickwit SLO or error budget breach detected on index
              {{ inputs.index_id }}. Canary query latency: {{
              (outputs.probe_canary_latency.elapsedTimeMicros / 1000) | int }}
              ms (SLA: {{ inputs.latency_sla_ms }} ms). Error log count: {{
              outputs.probe_error_volume.total }} (Budget: {{
              inputs.error_budget_count }} in last {{ inputs.lookback_minutes
              }}m).
    else:
      - id: log_healthy_status
        type: io.kestra.plugin.core.log.Log
        description: >
          Log normal operational status when search latency and error counts
          remain within acceptable thresholds.
        message: >
          HEALTHY: Quickwit index {{ inputs.index_id }} is operating within
          latency and error budget bounds. Canary latency: {{
          (outputs.probe_canary_latency.elapsedTimeMicros / 1000) | int }} ms.
          Error count in last {{ inputs.lookback_minutes }}m: {{
          outputs.probe_error_volume.total }}.

  - id: log_sentinel_audit_complete
    type: io.kestra.plugin.core.log.Log
    description: >
      Emit a final audit summary confirming the probe and evaluation cycle
      concluded.
    message: >
      Sentinel cycle complete for Quickwit index {{ inputs.index_id }} in
      execution {{ execution.id }}. Evaluated UID: {{
      outputs.probe_index_metadata.indexUid }}. Mode: {{ inputs.alert_mode }}.
      Breached: {{ (outputs.probe_canary_latency.elapsedTimeMicros >
      (inputs.latency_sla_ms * 1000)) or (outputs.probe_error_volume.total >
      inputs.error_budget_count) }}.

errors:
  - id: on_failure_alert
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    description: >
      Flow-level failure alert dispatched if the sentinel cannot reach Quickwit
      or encounters an unexpected task failure.
    url: "{{ secret('SLACK_WEBHOOK') }}"
    payload: |
      {
        "text": "Quickwit sentinel execution {{ execution.id }} FAILED for index {{ inputs.index_id }} in namespace {{ flow.namespace }}. Investigate node availability or network reachability immediately."
      }

Quickwit indexes petabytes of distributed logs, events, and OpenTelemetry traces into immutable splits stored on object storage. When data ingestion volume spikes, split counts fragment, or search worker nodes encounter resource contention, search query execution latency can silently degrade past acceptable service level agreements (SLAs), while application error bursts accumulate unnoticed.

This blueprint provides an automated Day 2 operations sentinel that inspects Quickwit index metadata, executes a canary search query to measure real-time execution duration in microseconds, audits recent error log document volume across a rolling lookback window, and triggers automated Slack alerts when operational budgets breach.

How it works

periodic_sentinel_schedule triggers the sentinel workflow on a 15-minute cron schedule (disabled by default so connection endpoints and thresholds can be configured).
probe_index_metadata calls io.kestra.plugin.quickwit.index.Get to retrieve index metadata, verifying the target index exists and reading its UID and configuration.
probe_error_volume invokes io.kestra.plugin.quickwit.search.Search with a query targeting severity:ERROR OR status:>=500 starting from now() - lookback_minutes. It inspects outputs.probe_error_volume.total to quantify recent error volume.
probe_canary_latency executes a wildcard canary query using io.kestra.plugin.quickwit.search.Search with fetchType: NONE to measure the raw search processing duration via outputs.probe_canary_latency.elapsedTimeMicros.
evaluate_slo_and_error_budget uses io.kestra.plugin.core.flow.If to check if canary latency in milliseconds exceeds latency_sla_ms or if the error document count exceeds error_budget_count.
When a breach occurs, evaluate_alert_mode checks alert_mode: ALERT posts a formatted Slack notification via io.kestra.plugin.slack.notifications.SlackIncomingWebhook, while AUDIT_ONLY logs the violation without paging on-call staff.
When performance is nominal, log_healthy_status records a healthy audit entry.
log_sentinel_audit_complete runs at the conclusion of every execution to maintain an unbroken audit trail of sentinel passes.
A flow-level errors handler catches connection errors or unreachable endpoints and alerts operators that the monitoring sentinel is offline.

What you get

Automated, continuous monitoring of Quickwit search query latency and error document spikes without deploying external monitoring daemons.
Dual-budget evaluation: covers both query responsiveness (latency SLA) and application health (log error budget).
Dry-run capability via AUDIT_ONLY mode to benchmark baseline cluster performance before enabling live alerting channels.
Out-of-band error handling that catches network partitions or cluster downtime and prevents silent monitoring failures.
Comprehensive execution metrics recorded directly in Kestra for auditability and compliance reporting.

Who it's for

Site Reliability Engineers (SREs) and DevOps engineers maintaining self-hosted Quickwit clusters for log management and trace storage.
Platform engineers responsible for search latency SLAs across enterprise observability stacks.
Data platform administrators seeking automated health gates and anomaly alerting on analytical search engines.

Why orchestrate this with Kestra

Quickwit does not provide native scheduled alerting or multi-channel notification dispatching out of the box. Kestra provides declarative, version-controlled workflow orchestration with built-in secret masking, conditional branching, native Slack integrations, and full execution history tracking. Managing the sentinel as a Kestra blueprint replaces fragile bash cron jobs with a resilient, observable automation pipeline.

Prerequisites

A running Quickwit instance (version 0.8.0 or later) reachable over HTTP/HTTPS from the Kestra worker environment.
A Quickwit index populated with searchable log or trace documents.
A Slack incoming webhook URL for dispatching alerts.

Secrets

SLACK_WEBHOOK: Incoming webhook URL for the target Slack channel (required for live alerting).

Note for Kestra Open-Source (OSS): Secrets are configured as base64-encoded environment variables formatted as SECRET_<NAME> (e.g. SECRET_SLACK_WEBHOOK).

Inputs

Name	Type	Default	Allowed Values	Description
`quickwit_url`	STRING	`"http://localhost:7280"`	Reachable Quickwit REST URL	Base URL of the Quickwit REST API service
`index_id`	STRING	`"app-logs"`	Valid index identifier	Identifier of the Quickwit index to audit
`lookback_minutes`	INT	`15`	Positive integer	Rolling time window in minutes to evaluate error volume
`latency_sla_ms`	INT	`500`	Positive integer	Maximum acceptable query duration in milliseconds
`error_budget_count`	INT	`25`	Positive integer	Maximum error documents allowed before triggering alert
`alert_mode`	SELECT	`"ALERT"`	`"ALERT"`, `"AUDIT_ONLY"`	Alerting behavior mode on threshold breach

Outputs

{{ outputs.probe_index_metadata.indexUid }}: Server-generated unique identifier of the target Quickwit index.
{{ outputs.probe_canary_latency.elapsedTimeMicros }}: Search query execution duration in microseconds.
{{ outputs.probe_error_volume.total }}: Total matching error log documents within the lookback window.

Quick start

Start a local Quickwit container on port 7280 (avoiding host port 8080):

docker run -d
--name quickwit-sentinel
-p 7280:7280
quickwit/quickwit:latest run
Connect the Quickwit container to the Kestra Docker network:

docker network connect YOUR_KESTRA_NETWORK quickwit-sentinel

Identify your network name using docker network ls.
Create a sample index and ingest sample documents following the official Quickwit tutorial:

curl -X POST http://localhost:7280/api/v1/indexes
-H "Content-Type: application/yaml"
--data-binary @tutorial-index-config.yaml
Configure the SLACK_WEBHOOK secret in your Kestra environment.
Import this blueprint into your Kestra instance.
Run the flow manually with alert_mode: AUDIT_ONLY to establish baseline search latency and verify connectivity.
Enable periodic_sentinel_schedule to begin automated 15-minute operational sweeps.

How to extend

Add automated index split compaction by triggering a POST request to /api/v1/indexes/{index_id}/compact when latency degrades.
Route alerts to PagerDuty or Microsoft Teams based on error severity using complementary notification plugins.
Integrate with a subflow to dynamically adjust ingest throttling or scale replica searcher nodes when latency remains elevated across consecutive runs.
Store historical latency and error counts in DuckDB or PostgreSQL for weekly SLA compliance reports.

Pitfalls

Port conflict: Quickwit runs on HTTP port 7280 by default. Ensure port 7280 is used in the JDBC/REST URL, and never bind container ports to 8080 to prevent collision with Kestra web UI.
Network isolation: When running Kestra and Quickwit in separate Docker containers, localhost:7280 will resolve to the Kestra container itself. Use the container hostname (e.g. http://quickwit-sentinel:7280) on a shared Docker bridge network.
Index naming: Quickwit enforces strict validation on index IDs (lowercase alphanumeric characters, underscores, and hyphens). Verify the index ID exists before activating schedules.
Large result payloads: When executing search queries, keep maxHits small (maxHits: 5) and use fetchType: NONE or fetchType: FETCH with limited limits to avoid exceeding Kestra memory limits.