Apify Actor Run Failure Sentinel
Audit an Apify actor's most recent run status, alert Slack when it failed, timed out, or was aborted, and optionally launch a fresh retry run under a cost cap.
Detect OpenSearch indices stuck read-only after the flood-stage disk watermark, confirm disk recovered, then clear the block and verify writes.
When a data node crosses the flood-stage disk watermark, OpenSearch sets index.blocks.read_only_allow_delete on its indices and every write returns HTTP 429. Freeing disk does not reliably bring writes back. On a single-node cluster the disk decider is off by default, so nothing re-evaluates the block. Plenty of production clusters disable the decider or sit right at the watermark. The block then stays until an operator runs PUT _all/_settings with the setting cleared to null. This blueprint does that safely: it finds the stuck indices, confirms disk has genuinely recovered, clears the block, re-reads the system to prove the clear landed then writes a probe document to prove writes are accepted again. It alerts Slack the moment a block is detected, before any remediation decision.
disk usage exceeded flood-stage watermark, index has read-only-allow-delete block, not 403. The io.kestra.plugin.opensearch.Request task fails on any non-2xx status, so a rejected call fails the run and the errors block alerts. Gates re-read the setting.null. Sending false leaves it in place.detect (io.kestra.plugin.opensearch.Request) reads GET {target}/_settings/index.blocks.read_only_allow_delete. OpenSearch returns one entry per blocked index and omits the rest, so the key count is the number of stuck indices.node_fs reads GET _nodes/stats/fs and the flow computes the busiest node's disk usage percent. cluster_settings reads the configured high watermark for the record.report logs the blocked indices, the measured usage and the mode. alert_detected then posts a Slack findings alert whenever a block is present, before and apart from the remediation gate, so a plain detect run or a dry_run run both alert.route (io.kestra.plugin.core.flow.If) acts only when a block is present, auto_remediate is true and dry_run is false. Otherwise report_only records a no-change outcome.disk_gate clears the block only when measured usage is under high_watermark_percent. If disk is still full it logs the refusal and fails so the alert fires.clear sends PUT {target}/_settings with read_only_allow_delete set to null. reverify and reverify_cluster re-read the system and landed fails the run if any index is still blocked.probe_write is true, probe writes one document and probe_landed fails unless the status is 200 or 201.errors block posts a Slack alert on any failure.Schedule trigger runs the scan every morning in report-only mode.Clearing a write block is a destructive-adjacent action that must be observed. Running it as a flow records which indices were stuck, what the disk looked like, whether the clear was allowed and whether writes came back, all against one execution id. The read-only default and the disk gate stop a careless clear during a real disk-full event. The re-read plus probe mean a green run actually means writes work.
Verified on Kestra 2.0.5 against OpenSearch 2.19.6 with security disabled. The flood-stage block was reproduced two ways: by lowering the watermark so the disk monitor set read_only_allow_delete on every index on its own and by applying the identical index setting the flood stage applies. A write while blocked returned HTTP 429 with the flood-stage reason. Clean, report, dry-run, remediate and refuse paths were all run and each outcome was confirmed with a direct OpenSearch call, not only the Kestra log.
OPENSEARCH_USERNAME: username for basic auth, when the cluster has security enabled.OPENSEARCH_PASSWORD: password for basic auth, when the cluster has security enabled.SLACK_WEBHOOK_URL: Slack incoming webhook URL for failure alerts.opensearch_url: base URL of the cluster.target: index or pattern to inspect and clear, _all by default.high_watermark_percent: disk usage ceiling before a clear is allowed, 90.0 by default.dry_run and auto_remediate: the two safety gates. A clear runs only when auto_remediate is true and dry_run is false.probe_write and probe_index: turn on a real write check after the clear.opensearch_url to your cluster. Run once with the defaults (dry_run true) to see which indices are blocked and what disk looks like.auto_remediate true and dry_run false to clear the block. Set probe_write true with a probe_index to confirm a real write.disabled: false on the daily trigger to scan every morning in report-only mode.target at a prefix like logs-* to scope the clear to one index family.high_watermark_percent to match your configured flood and high watermarks.This blueprint was created by zkasuran.
