Weekly OpenSearch Index Retention with Dated Deletes

Dated log indexes grow forever unless something deletes them on schedule. This blueprint enforces a thirty-day retention policy with io.kestra.plugin.opensearch.Request, the plugin's raw REST escape hatch: a DELETE call targets app-logs-yyyy.MM.dd where the date is computed by Pebble as thirty days before the run. Discord receives a confirmation naming the exact index and the HTTP status, so the retention trail is written where the team already reads.

How it works

1. delete_expired_index (io.kestra.plugin.opensearch.Request) sends DELETE to the endpoint app-logs-{{ (trigger.date ?? execution.startDate) | dateAdd(-30, 'DAYS') | date('yyyy.MM.dd') }}. Using trigger.date ?? execution.startDate means scheduled runs and manual backfills both compute the correct day.
2. The response status is exposed as {{ outputs.delete_expired_index.status }} and the body as {{ outputs.delete_expired_index.response }}.
3. confirm posts the deleted index name and the status code to Discord.
4. The errors block posts a distinct Discord alert when the delete fails, for example on a missing index or a permissions problem.
5. A disabled-by-default Schedule trigger enforces the policy every Monday at 03:00.

What you get

• Retention as code: the window, the index pattern, and the schedule live in one reviewable file.
• A Discord record of every deletion with the index name and HTTP status.
• The Request pattern for any OpenSearch admin API that has no dedicated task, aliases, settings, snapshots.
• A failure alert so a broken delete never turns into a full-disk incident weeks later.

Who it's for

• Platform teams running daily-dated log or event indexes who need predictable storage.
• Compliance-minded teams who must prove data older than the retention window is gone.
• Anyone replacing a curator container or a cron script nobody remembers deploying.

Why orchestrate this with Kestra

Deleting data is the one job that must be observable. Every run of this flow records which index it targeted, what the cluster answered, and who was told, all in the execution history. Backfills recompute the date correctly, failures alert a human, and changing the retention window is a one-line diff in version control instead of an edit inside a forgotten crontab.

Prerequisites

• Daily-dated indexes matching app-logs-yyyy.MM.dd, or adjust the endpoint pattern to your naming scheme.
• Credentials allowed to delete indexes; scope them to the pattern rather than granting cluster-wide admin.
• A Discord incoming webhook for confirmations and alerts.
• Note the weekly cadence deletes one dated index per run; run daily, or extend to a range, if you need every expired day removed.

Secrets

• OPENSEARCH_HOST: OpenSearch endpoint, for example https://opensearch.example.com:9200.
• OPENSEARCH_USERNAME: OpenSearch username.
• OPENSEARCH_PASSWORD: OpenSearch password.
• DISCORD_WEBHOOK_URL: Discord incoming webhook URL.

Quick start

1. Add the four secrets to your Kestra namespace.
2. Adjust the index pattern and the thirty-day window to your retention policy.
3. Execute the flow against a disposable test index dated thirty days back and check the Discord confirmation.
4. Set disabled: false on the weekly trigger.

How to extend

• Loop over several trailing days with io.kestra.plugin.core.flow.ForEach so one weekly run clears the whole expired range.
• Snapshot the index to object storage before deleting, using another Request call to the snapshot API.
• Export the index to internal storage first, as shown in the OpenSearch full index export blueprint.
• Apply the same pattern to other dated families, metrics, traces, audit events, by parameterizing the prefix as an input.

Links

• OpenSearch plugin documentation
• OpenSearch index API documentation