Schedule icon
Request icon
Log icon
If icon
SlackIncomingWebhook icon
Fail icon

Detect and Release OpenSearch Flood-Stage Write Blocks

Detect OpenSearch indices stuck read-only after the flood-stage disk watermark, confirm disk recovered, then clear the block and verify writes.

Categories
Infrastructure

When a data node crosses the flood-stage disk watermark, OpenSearch sets index.blocks.read_only_allow_delete on its indices and every write returns HTTP 429. Freeing disk does not reliably bring writes back. On a single-node cluster the disk decider is off by default, so nothing re-evaluates the block. Plenty of production clusters disable the decider or sit right at the watermark. The block then stays until an operator runs PUT _all/_settings with the setting cleared to null. This blueprint does that safely: it finds the stuck indices, confirms disk has genuinely recovered, clears the block, re-reads the system to prove the clear landed then writes a probe document to prove writes are accepted again. It alerts Slack the moment a block is detected, before any remediation decision.

Findings

  • The flood-stage block rejects writes with HTTP 429 and the reason disk usage exceeded flood-stage watermark, index has read-only-allow-delete block, not 403. The io.kestra.plugin.opensearch.Request task fails on any non-2xx status, so a rejected call fails the run and the errors block alerts. Gates re-read the setting.
  • The block clears only with the body value null. Sending false leaves it in place.
  • Clearing while disk is still full just re-trips the flood stage, so the flow refuses to clear unless measured disk usage is under the high watermark.

How it works

  1. detect (io.kestra.plugin.opensearch.Request) reads GET {target}/_settings/index.blocks.read_only_allow_delete. OpenSearch returns one entry per blocked index and omits the rest, so the key count is the number of stuck indices.
  2. node_fs reads GET _nodes/stats/fs and the flow computes the busiest node's disk usage percent. cluster_settings reads the configured high watermark for the record.
  3. report logs the blocked indices, the measured usage and the mode. alert_detected then posts a Slack findings alert whenever a block is present, before and apart from the remediation gate, so a plain detect run or a dry_run run both alert.
  4. route (io.kestra.plugin.core.flow.If) acts only when a block is present, auto_remediate is true and dry_run is false. Otherwise report_only records a no-change outcome.
  5. disk_gate clears the block only when measured usage is under high_watermark_percent. If disk is still full it logs the refusal and fails so the alert fires.
  6. clear sends PUT {target}/_settings with read_only_allow_delete set to null. reverify and reverify_cluster re-read the system and landed fails the run if any index is still blocked.
  7. When probe_write is true, probe writes one document and probe_landed fails unless the status is 200 or 201.
  8. The errors block posts a Slack alert on any failure.
  9. A disabled Schedule trigger runs the scan every morning in report-only mode.

What you get

  • A one-command recovery for the most common OpenSearch disk incident, driven through the plugin and logged as an execution.
  • A safety gate that refuses to clear while disk is still full, so the block does not re-trip the moment it is removed.
  • Proof the fix landed, from a re-read of the setting and an optional real write, not from a hopeful 200.
  • A Slack findings alert the moment a block is detected, independent of whether a clear will run.
  • A Slack failure alert whenever the block persists after a clear or the clear is refused.

Who it's for

  • Platform and SRE teams whose ingest stops with HTTP 429 after a disk scare and stays stopped.
  • Teams running single-node or small OpenSearch clusters where the write block never auto-clears.
  • Anyone who has cleared this block by hand at 3am and wants it gated, logged and alertable instead.

Why orchestrate this with Kestra

Clearing a write block is a destructive-adjacent action that must be observed. Running it as a flow records which indices were stuck, what the disk looked like, whether the clear was allowed and whether writes came back, all against one execution id. The read-only default and the disk gate stop a careless clear during a real disk-full event. The re-read plus probe mean a green run actually means writes work.

Tested end to end

Verified on Kestra 2.0.5 against OpenSearch 2.19.6 with security disabled. The flood-stage block was reproduced two ways: by lowering the watermark so the disk monitor set read_only_allow_delete on every index on its own and by applying the identical index setting the flood stage applies. A write while blocked returned HTTP 429 with the flood-stage reason. Clean, report, dry-run, remediate and refuse paths were all run and each outcome was confirmed with a direct OpenSearch call, not only the Kestra log.

Prerequisites

  • A reachable OpenSearch cluster. For a secured cluster, credentials allowed to read settings and node stats and to update index settings.
  • A Slack incoming webhook for failure alerts.
  • For the probe write, an existing index the flow may append a one-document marker to.

Secrets

  • OPENSEARCH_USERNAME: username for basic auth, when the cluster has security enabled.
  • OPENSEARCH_PASSWORD: password for basic auth, when the cluster has security enabled.
  • SLACK_WEBHOOK_URL: Slack incoming webhook URL for failure alerts.

Inputs

  • opensearch_url: base URL of the cluster.
  • target: index or pattern to inspect and clear, _all by default.
  • high_watermark_percent: disk usage ceiling before a clear is allowed, 90.0 by default.
  • dry_run and auto_remediate: the two safety gates. A clear runs only when auto_remediate is true and dry_run is false.
  • probe_write and probe_index: turn on a real write check after the clear.

Quick start

  1. Add the Slack webhook secret plus the OpenSearch credentials if your cluster is secured.
  2. Set opensearch_url to your cluster. Run once with the defaults (dry_run true) to see which indices are blocked and what disk looks like.
  3. When disk has recovered, run with auto_remediate true and dry_run false to clear the block. Set probe_write true with a probe_index to confirm a real write.
  4. Set disabled: false on the daily trigger to scan every morning in report-only mode.

How to extend

  • Point target at a prefix like logs-* to scope the clear to one index family.
  • Raise or lower high_watermark_percent to match your configured flood and high watermarks.
  • Add a second alert branch that pages when the clear is refused because disk is genuinely full, which is a capacity problem the clear cannot fix.
  • Chain a downstream flow off this flow's FAILED state to open a ticket when a block cannot be cleared.

Links

This blueprint was created by zkasuran.

See How

New to Kestra?

Use blueprints to kickstart your first workflows.