OpenSearch ISM Policy Compliance Gate

An index created without an Index State Management (ISM) policy has no rollover, no retention, and no deletion lifecycle - it just grows until disk pressure or a cluster health alert forces someone to deal with it by hand. This blueprint reads OpenSearch's own ISM Explain API to check whether a target index actually has a policy attached, alerts on a gap or on an index that could not be reached, and can attach the expected policy with OpenSearch's own ISM Add Policy API - which, per OpenSearch's own documentation, never overwrites a policy an index already has.

How it works

1. periodic_policy_check (io.kestra.plugin.core.trigger.Schedule) runs every 6 hours, always forcing auto_remediate: "false" and dry_run: "true" through its own inputs: override regardless of this flow's defaults. Shipped disabled so you can validate a manual run first.
2. check_ism_status (io.kestra.plugin.opensearch.Request, GET _plugins/_ism/explain/{{ index_name }}) returns status (HTTP code) and response (the Explain API's JSON body, keyed by index name).
3. evaluate_index_reachability (io.kestra.plugin.core.flow.If) branches on status == 200.
   • else (UNREACHABLE): alert_index_unreachable reports the non-200 status - the index may not exist, or ISM may be unavailable.
   • then (index reachable): nested evaluate_policy_attachment (io.kestra.plugin.core.flow.If) branches on whether response["{{ index_name }}"].policy_id, read with a dynamically built jq filter, is non-null.
     • then (HEALTHY): log_policy_attached records the attached policy_id.
     • else (AT_RISK): route_remediation (io.kestra.plugin.core.flow.Switch on auto_remediate) either considers remediation ("true", gated again by dry_run through check_dry_run) or only logs that remediation is disabled ("false"). When both gates allow it, attach_ism_policy calls POST _plugins/_ism/add/{{ index_name }} with {"policy_id": expected_policy_id}. alert_policy_gap always fires to Slack regardless of which remediation path ran.
4. log_audit_complete always runs last, printing the headline numbers regardless of which branch fired.
5. The errors block alerts Slack separately if the flow itself fails outright - an unreachable cluster or bad credentials must not read as "no gap".

What you get

• A direct answer, from OpenSearch's own ISM Explain API, to "does this index actually have a lifecycle policy attached" - not inferred from index naming conventions or age.
• A distinct UNREACHABLE state so an index that does not exist (or an ISM plugin that is unavailable) is never conflated with either a healthy or an at-risk index.
• Remediation that relies on OpenSearch's own documented behavior: attaching a policy to an already-managed index does not overwrite it, it is reported as a failed update instead.
• Two independent gates (auto_remediate, dry_run) between "gap detected" and "policy actually attached".
• A final status log every run, gap or not, so the execution history doubles as an ISM-compliance trend for that index.

Who it's for

• Platform and search infrastructure teams running OpenSearch who need an early signal when a newly created index was never given a retention or rollover policy.
• Teams with naming or provisioning conventions (like app-logs-*) where policy attachment is expected but not otherwise enforced.
• Anyone evaluating the OpenSearch plugin who wants a worked example of Request driving both a management-plane read (_ism/explain) and a management-plane write (_ism/add), not just document indexing or search.

Why orchestrate this with Kestra

The ISM Explain API run by hand answers one moment in time; it does not decide a cadence, classify the result into a three-state risk level, gate a remediation behind two independent confirmations, or notify anyone. Kestra supplies the schedule, the HEALTHY/AT_RISK/UNREACHABLE classification as a first-class branch, two independent authorization gates before any policy is attached, and an execution history that shows exactly when an index went ungoverned and whether it was fixed.

Prerequisites

• A reachable OpenSearch cluster with the Index State Management (ISM) plugin available. ISM ships with the official opensearchproject/opensearch distribution as a core plugin, unlike the optional opensearch-sql plugin some other blueprints in this repo depend on - but this has not been confirmed against a live container in this environment, so treat it as unverified until you run it.
• An ISM policy already created under expected_policy_id (via PUT _plugins/_ism/policies/{policy_id}, not part of this flow) before auto_remediate is ever set to "true" - the Add Policy API attaches an existing policy by ID, it does not create one.
• A Slack incoming webhook for alerts.

Local testing: docker run -d --name opensearch-ism-gate -p 9202:9200 -p 9602:9600 -e "discovery.type=single-node" -e "DISABLE_SECURITY_PLUGIN=true" opensearchproject/opensearch:2.11.0 docker network connect YOUR_KESTRA_NETWORK opensearch-ism-gate (only needed if the Kestra Worker runs in a separate Docker network than this container; set OPENSEARCH_HOST to http://opensearch-ism-gate:9200 from inside that network, or http://localhost:9202 from the host). With DISABLE_SECURITY_PLUGIN=true, OPENSEARCH_USERNAME/OPENSEARCH_PASSWORD can be set to any placeholder value - the cluster does not check them.

Secrets

• OPENSEARCH_HOST: OpenSearch connection.hosts entry, scheme and port included.
• OPENSEARCH_USERNAME / OPENSEARCH_PASSWORD: basicAuth credentials used by every io.kestra.plugin.opensearch.Request task in this flow.
• SLACK_WEBHOOK_URL: Slack incoming webhook used by alert_policy_gap, alert_index_unreachable, and the errors block.
• In Kestra OSS (no Enterprise secrets backend), secrets are supplied as environment variables prefixed SECRET_, base64-encoded, and read back in flows with {{ secret('NAME') }} - for example SECRET_OPENSEARCH_PASSWORD=$(echo -n '<your-local-password>' | base64). This keeps credentials out of the flow YAML but, per Kestra's own documentation, offers no encryption at rest or access control beyond the host environment; use the Enterprise secrets backend for stronger guarantees.

Inputs

• index_name (STRING, default app-logs): index audited and, if remediated, updated.
• expected_policy_id (STRING, default app-logs-policy): policy_id expected to be attached, and applied on remediation.
• auto_remediate (SELECT: "false", "true"; default "false"): must be "true" for remediation to even be considered.
• dry_run (SELECT: "true", "false"; default "true"): must be explicitly "false", together with auto_remediate: "true", for the ISM Add Policy API to actually run.

Outputs

• outputs.check_ism_status.status / .response: the ISM Explain API's raw HTTP status and JSON body, every run.
• outputs.attach_ism_policy.status / .response: the ISM Add Policy API's raw HTTP status and JSON body, present only when remediation actually ran.

Quick start

1. Start OpenSearch locally with the command above, create an index with no policy (PUT app-logs), and separately create a policy under app-logs-policy with PUT _plugins/_ism/policies/app-logs-policy.
2. Add the OPENSEARCH_HOST, OPENSEARCH_USERNAME, OPENSEARCH_PASSWORD, and SLACK_WEBHOOK_URL secrets.
3. Run the flow manually with the defaults and confirm alert_policy_gap fires (AT_RISK: the index exists, no policy is attached yet).
4. Run once with auto_remediate: "true" and dry_run: "true" and confirm log_dry_run_remediation describes the attachment without calling the API.
5. Run once with auto_remediate: "true" and dry_run: "false" and confirm attach_ism_policy runs, then verify with GET _plugins/_ism/explain/app-logs that policy_id is now set.
6. Point index_name at a non-existent index and re-run to confirm alert_index_unreachable fires (UNREACHABLE) instead of either HEALTHY or AT_RISK.
7. Enable periodic_policy_check once you trust the check; it always runs in the safe auto_remediate: "false" / dry_run: "true" mode regardless of what you leave the flow's own defaults set to.

How to extend

• Loop over several index_name values (or an index pattern resolved with a prior Request to _cat/indices) with io.kestra.plugin.core.flow.Loop to run the same gate fleet-wide in one flow.
• Add a second check on the Explain response's enabled field to catch a policy that is attached but disabled, a different risk than no policy at all.
• Use the show_policy=true query parameter on the Explain API to compare the currently applied policy's content against the latest version from the Get Policy API, catching stale-policy drift in addition to missing-policy drift.
• Route the UNREACHABLE branch to a different Slack channel than AT_RISK, since a missing index often means an upstream ingestion pipeline never created it, not a governance gap.

Pitfalls

• UNREACHABLE here means "Explain API did not return HTTP 200", not "cluster is down". A fully down cluster fails the Request task's own connection and is caught by the errors block, not by evaluate_index_reachability - that branch only fires when the cluster answers but with a non-200 status, most commonly a 404 for a nonexistent index.
• There is no structured "ISM" task in this plugin. Verified from the plugin's own source (AbstractLoad, AbstractSearch, AbstractTask, Bulk, Get, Load, Ppl, Put, Request, Scroll, Search): both the Explain read and the Add Policy write go through the same generic io.kestra.plugin.opensearch.Request task used elsewhere in this repo for _update_by_query.
• Attaching a policy never overwrites an existing one. Per OpenSearch's own ISM API documentation, the Add Policy API returns HTTP 200 even when the index already has a policy, but reports the index in failed_indices with failures: true instead of actually changing anything - attach_ism_policy's own response output carries this, nothing in this flow parses or acts on failed_indices differently, so check outputs.attach_ism_policy.response directly if you need to confirm the attach actually took.
• auto_remediate and dry_run are independent gates, not a single boolean. Both must be "true"/"false" respectively for the Add Policy API to run; setting only one leaves the other still blocking.
• Switch case keys "true"/"false" are quoted strings. auto_remediate renders as the literal string "true" or "false"; unquoted true:/false: YAML map keys would parse as booleans instead and would not match.
• This blueprint is UNTESTED against a live OpenSearch cluster. The _plugins/_ism/explain/{index} and _plugins/_ism/add/{index} endpoints and their exact JSON shapes are taken verbatim from OpenSearch's own ISM API documentation, and the Request task's properties (method, endpoint, parameters, body, connection) and outputs (status, response) are taken verbatim from the plugin's source on main - but no Docker container or Kestra engine was run to execute this flow end to end.

Links

• plugin-opensearch documentation
• plugin-opensearch source (Request.java)
• OpenSearch ISM API reference
• OpenSearch Index State Management overview
• Kestra secrets (OSS vs Enterprise)