Schedule icon
Request icon
If icon
Fail icon
OutputValues icon
Log icon
SlackIncomingWebhook icon
Pause icon
Loop icon
LoopUntil icon

Cancel Runaway Elasticsearch Reindex and By-Query Tasks

Detect Elasticsearch reindex and by-query tasks running past a threshold, cancel them behind an approval gate, then verify the task list is clear.

Categories
Infrastructure

A _reindex, _update_by_query or _delete_by_query launched with wait_for_completion=false returns a task id and then runs with nobody watching it. A badly scoped one saturates the cluster for hours. Search latency climbs, merges back up, the write queue grows. No circuit breaker fires, because from the cluster's point of view it is doing work it was asked to do. The caller who started it has usually gone home with the task id in a terminal that is now closed. This blueprint lists the tasks in those action families, flags the ones whose running_time_in_nanos is past a threshold, cancels them behind a dry run and a human approval, waits out the cooperative cancellation window, then re-reads _tasks and fails the run if a targeted task is still there.

Safety

  • The action allowlist is the safety boundary, not a convenience. Elasticsearch runs internal tasks that keep the cluster alive, so the flow never cancels an arbitrary task id. task_actions is matched exactly against each task's action, which means a wildcard typed into it matches no action at all instead of widening the blast radius. The default list is the reindex family plus the two by-query families.
  • Cancellation is cooperative. POST _tasks/{id}/_cancel sets a flag and returns straight away. The task keeps draining until it reaches its next checkpoint, so it can legitimately still be in the task list for a short window afterwards. The flow waits that out in a bounded LoopUntil before it re-reads, then reports whether a task that is still present has its cancel flag set (draining) or not (the cancel never took).
  • A cancelled _update_by_query or _delete_by_query does not roll back. Every document it already updated or deleted stays that way. The index is left part way through the operation. That is why the cancel sits behind a Pause: the operator needs to decide what to do with a half-applied batch. The flow says so in its own log rather than reporting a clean stop.
  • max_cancel_per_run refuses the pass when the flagged list is long, because that usually means the threshold is below the normal runtime of this cluster's jobs rather than that the cluster is full of runaway work.
  • A child task of a sliced reindex is dropped from the list. Cancelling the parent takes its slices with it, so cancelling each slice as well is noise.

Findings

  • io.kestra.plugin.elasticsearch.Request reports task success on any HTTP status, so a 404 on a mistyped endpoint looks exactly like an idle cluster. Every read here is gated on outputs.<id>.status. The final verdict comes from a re-read of _tasks rather than from task state.
  • The same task deserializes the response body into a map, so any _cat endpoint with format=json fails the task outright on its JSON array. _tasks is read with group_by=none, which keeps the response a map with one flat tasks array inside it.
  • A task's description is Elasticsearch's own rendering of the query or script that started it, so it carries quotes, braces and sometimes newlines. The Slack payload is built as a map and serialised with toJson, so that text cannot break out of the JSON string it sits in.
  • _tasks with a filter_path and nothing running returns an empty object, not an empty array, so every jq program here reads (.tasks) // [].

How it works

  1. detect_cluster (io.kestra.plugin.elasticsearch.Request) reads GET _cluster/health for the status, the pending cluster task count and the queue wait, which is the context a long running task shows up in.
  2. detect_tasks reads GET _tasks with actions set to the allowlist, detailed=true and group_by=none, so each entry carries its node id, its running time, its cancellable flag and its by-query status counters.
  3. check_reads (io.kestra.plugin.core.flow.If) fails the run when either read did not return 200.
  4. plan (io.kestra.plugin.core.output.OutputValues) keeps the cancellable top-level tasks whose action is in the allowlist by exact match and whose running_time_in_nanos is past max_running_seconds.
  5. report logs the cluster context, the flagged tasks and the mode. alert_detected posts a Slack findings alert whenever something is flagged, before and apart from the cancel decision, so a plain scan or a dry run both alert.
  6. route acts only when something is flagged, auto_remediate is true and dry_run is false. cap_gate then refuses unless the flagged list is at or under max_cancel_per_run, logging the refusal and failing so the alert fires.
  7. approval_gate (io.kestra.plugin.core.flow.Pause) holds the run until a human resumes it with approved set true. remediate_cancel_tasks (io.kestra.plugin.core.flow.Loop, concurrencyLimit: 1) then sends one POST _tasks/{task_id}/_cancel per flagged task.
  8. settle (io.kestra.plugin.core.flow.LoopUntil) polls _tasks every five seconds until no targeted task id is left or settle_timeout runs out. It never fails on its own, it only bounds the wait.
  9. reverify_tasks re-reads _tasks. verify_result fails the run when any targeted task id is still in the list. The failure message distinguishes a task that is still draining from one whose cancel was never accepted.
  10. The errors block posts a Slack alert on any failure. A disabled Schedule trigger runs the scan every hour in report-only mode.

What you get

  • A named list of the reindex and by-query tasks that are actually running right now, with how long each has been going and how many batches it has done.
  • A threshold in seconds instead of a guess, so a nightly reindex that normally takes 20 minutes is never touched.
  • Cancellation that is proven, not assumed, because the run goes red when a targeted task is still in the task list.
  • An allowlist that cannot be widened by accident, so an internal cluster task is never a candidate.
  • A refusal to act when the flagged list is long, which is the shape of a wrong threshold.
  • Slack on detection and Slack on failure, both through a secret.

Who it's for

  • Platform and SRE teams whose cluster goes slow at unpredictable times with no deploy to blame.
  • Anyone running bulk reindex or by-query maintenance from a terminal, where wait_for_completion=false is the normal way to start it.
  • Teams who want a scheduled scan that reports long running tasks every hour without being able to cancel anything until somebody approves it.

Why orchestrate this with Kestra

Cancelling a by-query task is a decision with consequences, because the documents it already changed stay changed. That needs a record of what was running, how long it had been going, who approved stopping it and whether it actually stopped. A flow gives all of that against one execution id. The read-only default, the max_cancel_per_run cap and the Pause stop a careless cancel of work somebody is waiting for. The bounded settle wait plus the re-read is what turns the run green or red rather than the cancel call returning 200.

Tested end to end

Verified on Kestra 2.0.5 against Elasticsearch 8.15.3 with security disabled, on a single node. A 600 document index was given a deliberately throttled POST /_update_by_query?wait_for_completion=false&requests_per_second=1&scroll_size=1, which runs for about ten minutes and shows up in _tasks with a growing running_time_in_nanos. The clean, detect, dry-run, approved-cancel, declined-approval, refused-by-cap and re-verify-fails paths were all run. Every number was confirmed with a direct call to the Elasticsearch _tasks API rather than read off the Kestra log. The detector was proven to report zero with no long running task, exactly one task with the runaway present. Zero again with the threshold set above the task's runtime. Shortening settle_timeout below the time the task needs to drain made the re-verify step fail the run as designed.

Prerequisites

  • A reachable Elasticsearch cluster. For a secured cluster, credentials allowed to read cluster health and the task list and to cancel tasks (the monitor plus manage cluster privileges).
  • A Slack incoming webhook for the findings and failure alerts.

Secrets

  • ELASTICSEARCH_USERNAME: username for basic auth, when the cluster has security enabled.
  • ELASTICSEARCH_PASSWORD: password for basic auth, when the cluster has security enabled.
  • SLACK_WEBHOOK_URL: Slack incoming webhook URL for alerts.

Inputs

  • elasticsearch_url: base URL of the cluster.
  • task_actions: comma separated allowlist of cancellable action names, matched exactly. Defaults to the reindex family plus both by-query families.
  • max_running_seconds: a task is flagged once it has been running longer than this.
  • max_cancel_per_run: the flow refuses to cancel anything when more tasks than this are flagged.
  • settle_timeout: how long to wait for cancelled tasks to leave the task list before the re-read.
  • dry_run and auto_remediate: the two safety gates. A cancel runs only when auto_remediate is true and dry_run is false, then only after the Pause is resumed with approved true.

Quick start

  1. Add the Slack webhook secret plus the Elasticsearch credentials if your cluster is secured.
  2. Set elasticsearch_url and run once with the defaults to see which tasks are running and how long they have been going.
  3. Set max_running_seconds above the normal runtime of your longest scheduled reindex.
  4. Run with auto_remediate true and dry_run false, then resume the Pause with approved true to cancel the flagged tasks.
  5. Set disabled: false on the hourly trigger to scan every hour in report-only mode.

How to extend

  • Narrow task_actions to one family, for example only indices:data/write/delete/byquery, to guard the most destructive one on a tighter threshold.
  • Add a second scheduled run with a much lower max_running_seconds in report-only mode, so you get an early warning long before anything is a candidate for cancelling.
  • Chain a flow off this flow's FAILED state to open a ticket, since a task that outlives the settle window with its cancel flag set is usually blocked on something other than the cancel.
  • Follow a cancelled _update_by_query with a flow that counts the documents the script already touched, so the half-applied batch is measured rather than guessed at.

Links

This blueprint was created by zkasuran.

See How

New to Kestra?

Use blueprints to kickstart your first workflows.