List icon
OutputValues icon
If icon
SlackIncomingWebhook icon
Log icon
Schedule icon

Quickwit Delete Task Backlog Sentinel

Audit a Quickwit index's queued delete tasks, alert Slack on a growing backlog, and log the result for data platform teams.

Categories
Data

A Quickwit delete task stays listed on its index until the background merge pipeline actually removes the matching documents; if that pipeline falls behind, the queue grows and nothing in Quickwit itself pages anyone about it. This blueprint reads the real queued delete tasks and their real creation timestamps, and reports the exact backlog, not a guess from a dashboard someone has to remember to check.

This blueprint was created by Mohit Jeswani.

How it works

  1. list_delete_tasks (io.kestra.plugin.quickwit.deletetask.List) lists every delete task still queued on index_id. Read-only; nothing in Quickwit is changed. allowFailure: true covers an unreachable Quickwit instance or a missing index without failing the flow outright.
  2. evaluate (io.kestra.plugin.core.output.OutputValues) reads the real queued count directly from the response's own size field, and computes the oldest task's age from its real createTimestamp with a jq filter.
  3. route (io.kestra.plugin.core.flow.If):
    • compliant: logs the queue depth, within limits.
    • alert: posts the queued count and the oldest task's age to Slack.
  4. summary logs both numbers and the severity, closing the governance cycle.
  5. errors alerts Slack if the flow itself fails, for example an unreachable Quickwit cluster.

What you get

  • A queue-depth and age check built on Quickwit's own delete-task bookkeeping, not an estimate from watching index size over time.
  • Both a count ceiling and an age ceiling in one check, since a backlog can grow either by volume or by one stuck task sitting for a long time.
  • A read-only check with zero risk of creating or cancelling a real delete task.

Who it's for

  • Data platform and search infrastructure teams running Quickwit who need to know when the deletion pipeline is falling behind, not discover it when disk usage keeps climbing despite delete requests.
  • Teams evaluating the Quickwit plugin who want a worked example of deletetask.List used as a standing backlog sentinel, distinct from this repo's existing Quickwit blueprints, which audit index source connectors and search latency rather than queued delete tasks.

Why orchestrate this with Kestra

Quickwit lists its queued delete tasks on request; it does not keep a schedule, compare the queue depth or oldest task's age against a tolerance you declare, or notify a Slack channel when either crosses it. Kestra keeps max_queued_count and max_task_age_seconds as flow inputs, turns the list into a Slack page naming the real numbers, and keeps a full execution history of every check.

Prerequisites

  • A Quickwit cluster, locally testable via the official quickwit/quickwit Docker image, with an index that has at least one delete task issued against it.
  • A Slack incoming webhook URL.

Secrets

  • SLACK_WEBHOOK_URL: used by alert and the errors block. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_SLACK_WEBHOOK_URL.

Inputs

Input Default Purpose
index_id <your-quickwit-index> Index to audit
max_queued_count 10 Queued delete task count at or above which the alert fires
max_task_age_seconds 3600 Oldest queued task's age at or above which the alert fires

Outputs

  • {{ outputs.list_delete_tasks.deleteTasks }} / .size: every queued delete task (createTimestamp, opstamp, deleteQuery) and how many.
  • {{ outputs.evaluate.values.queued_count }} / .oldest_age_seconds: the same count, and how old the oldest queued task is, in seconds.
  • {{ outputs.evaluate.values.severity }}: OK, DELETE_TASK_BACKLOG, or CRITICAL.

Pitfalls

  • A delete task remains listed until Quickwit's merge pipeline actually applies it; a healthy index can briefly show a nonzero queue between a delete request and the next merge cycle, which is why both a count and an age threshold exist rather than alerting on any nonzero queue at all.
  • deleteTasks has no explicit per-task status field; this check infers backlog purely from count and age, not from any "in progress" versus "pending" distinction Quickwit itself does not expose through this endpoint.
  • index_id must be an existing index; a typo surfaces via allowFailure as CRITICAL, not a silently empty queue.

Quick start

  1. Run Quickwit locally (docker run -p 7280:7280 quickwit/quickwit:latest run) and add the SLACK_WEBHOOK_URL secret.
  2. Set index_id to a real index, issue a delete task against it via Quickwit's own API.
  3. Run the flow manually and confirm summary logs the queued count and oldest age.
  4. Lower max_queued_count or max_task_age_seconds below the real current values to confirm alert fires.
  5. Enable the hourly trigger once both thresholds reflect your real operational policy.

How to extend

  • Loop over several index_id values with io.kestra.plugin.core.flow.Loop to audit every index in a cluster in one scheduled run.
  • Include each queued task's deleteQuery.query in the Slack alert to show exactly which documents are waiting to be removed.
  • Track queued_count over successive runs in a KV entry to chart whether the backlog is growing or shrinking, not just its current size.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.