Schedule icon
Get icon
Search icon
Log icon
Switch icon
SlackIncomingWebhook icon

Quickwit Index Retention Policy and Cold Storage Headroom Sentinel

Audit Quickwit search indexes for missing retention policies and split accumulation, verify canary query liveness, and alert Slack on storage quota breaches.

Categories
Data

Proactively audit Quickwit cloud-native search indexes to detect unconfigured or non-compliant document retention policies, verify search engine responsiveness via canary queries, and dispatch actionable Slack alerts before storage bloat or compliance violations occur.

How it works

  1. Automated Surveillance Trigger: periodic_retention_audit_schedule (io.kestra.plugin.core.trigger.Schedule) initiates a periodic retention audit every 6 hours, with on-demand manual runs supported.
  2. Multi-Stage Cluster Probes:
    • probe_index_metadata (io.kestra.plugin.quickwit.index.Get) retrieves the effective configuration for index_id, inspecting document mappings, source checkpoints, and the retention specification block.
    • probe_search_availability (io.kestra.plugin.quickwit.search.Search) executes a lightweight canary search query (*) to verify query service liveness and capture query execution latency in microseconds.
  3. Pebble Compliance Evaluation: Evaluates whether a retention block exists and checks if retention.period aligns with organizational data governance rules (approved_retention_period), classifying index state as COMPLIANT, POLICY_RETENTION_EXCEEDED, or CRITICAL_RETENTION_UNCONFIGURED.
  4. Switch Routing & Safe Remediation:
    • CRITICAL_RETENTION_UNCONFIGURED: Emits an error log and dispatches a high-priority Slack incident alert with index UID and actionable guidance on adding a daily retention schedule.
    • POLICY_RETENTION_EXCEEDED: Emits a warning log and dispatches a Slack warning notification detailing the retention period drift.
    • COMPLIANT: Emits an operational verification log confirming retention schedules and search latency are within SLA bounds.
  5. Audit Trail Logging: log_audit_summary (io.kestra.plugin.core.log.Log) records a persistent telemetry summary in Kestra execution logs for compliance audits and historical capacity tracking.
  6. Error Incident Fallback: Dedicated errors tasks catch cluster unreachable errors or HTTP timeouts, notifying on-call engineers immediately.

What you get

  • Automated retention policy governance across self-hosted or cloud Quickwit search clusters.
  • Early warning detection for missing retention policies that risk runaway cloud object storage (S3/MinIO/GCS) bills.
  • Real-time Slack notifications with rich block formatting and remediation recommendations.
  • Integrated canary query liveness testing to continuously validate search API availability.
  • Fully testable against standard open-source Quickwit Docker images.

Who it's for

  • Data Platform & FinOps Engineers: Preventing cloud storage cost overruns caused by unpruned index split files.
  • Site Reliability Engineers (SREs): Automating log cluster health monitoring and verifying search query response times.
  • Security & Compliance Officers: Enforcing mandatory data retention limits and GDPR log sanitization schedules.

Why orchestrate this with Kestra

Quickwit delegates storage garbage collection to an internal janitor loop, but cannot alert external teams if an index author forgets to define a retention block. Orchestrating this sentinel with Kestra provides declarative YAML workflows, secret isolation for webhook credentials, built-in retry handling for cluster maintenance windows, and comprehensive audit history.

Prerequisites

  • A running Quickwit cluster reachable from the Kestra worker over HTTP (default port 7280).
  • At least one active Quickwit index created on the cluster.
  • An incoming Slack webhook URL for alert notifications.

Secrets

  • SLACK_WEBHOOK_URL: Slack Incoming Webhook URL used to dispatch retention incident alerts.

Quick start

  1. Start a local open-source Quickwit container for testing:
    docker run -d --name quickwit -p 7280:7280 quickwit/quickwit:latest run
    
  2. Add the SLACK_WEBHOOK_URL secret in your Kestra namespace.
  3. Import this blueprint into your Kestra instance.
  4. Execute the flow manually with default inputs to inspect the retention audit results.
  5. Enable the periodic_retention_audit_schedule trigger for automated background monitoring.

How to extend

  • Automated Retention Patching: Add an io.kestra.plugin.quickwit.index.Create task in the remediation branch with docMapping and retention configured to automatically patch unconfigured indexes.
  • Multi-Index Fleet Audit: Use io.kestra.plugin.quickwit.index.List combined with a Loop flow task to iterate through and audit every index in the cluster in a single run.
  • Prometheus Metrics Export: Push canary latency and retention compliance gauges to an OpenTelemetry or Prometheus Pushgateway.
  • PagerDuty Escalation: Route critical missing-retention alerts to PagerDuty or Opsgenie during production deployments.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.