Full OpenSearch Index Export to Internal Storage with Scroll

A plain search caps out at the result window; archiving an index means walking all of it. This blueprint uses io.kestra.plugin.opensearch.Scroll, which pages through the complete result set behind one task, with the scroll export, so every document streams into a file in Kestra's internal storage instead of worker memory. Slack gets the row count when the export lands, and the file itself stays attached to the execution.

How it works

1. export_index (io.kestra.plugin.opensearch.Scroll) runs a match_all query against the app-logs index. The scroll API pages through every hit, and the scroll export writes the rows to internal storage, exposing {{ outputs.export_index.uri }} for downstream tasks and {{ outputs.export_index.size }} as the row count.
2. notify posts the row count to Slack and notes that the file stays on the execution, in the Outputs tab, so nobody hunts for a path.
3. The errors block posts a distinct Slack alert on failure, making a missing archive explicit.
4. A disabled-by-default Schedule trigger runs the export on the first of every month at 05:00.

What you get

• A complete index export in one declarative task, no scroll cursor bookkeeping in client code.
• Memory-safe archiving: documents stream to a file, so index size does not dictate worker size.
• The export addressable as a URI output, ready to hand to any downstream load or upload task.
• A Slack record of every archive with its row count, plus an alert when one fails.

Who it's for

• Platform teams archiving log indexes before a retention job deletes them.
• Data engineers moving OpenSearch data toward a lake or warehouse and needing a faithful extract first.
• Compliance teams who must keep point-in-time copies of searchable records.

Why orchestrate this with Kestra

An export script has to manage scroll cursors, buffer to disk, land the file somewhere durable, and report completion; this flow declares all four. Internal storage ties the artifact to the execution that produced it, so every archive is traceable to a date, a query, and a row count. Rerunning a failed month is one click, and downstream flows can consume the URI output without knowing how it was produced.

Prerequisites

• An OpenSearch index to archive, app-logs by default.
• Credentials allowed to search and scroll that index.
• A Slack incoming webhook for reports and alerts.
• Internal storage sized for the export, roughly the index's primary store size.

Secrets

• OPENSEARCH_HOST: OpenSearch endpoint, for example https://opensearch.example.com:9200.
• OPENSEARCH_USERNAME: OpenSearch username.
• OPENSEARCH_PASSWORD: OpenSearch password.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the four secrets to your Kestra namespace.
2. Point indexes at the index you want to archive.
3. Execute the flow, check the Slack row count, and open the execution's Outputs tab to find the stored file.
4. Set disabled: false on the monthly trigger.

How to extend

• Narrow the query from match_all to a range on @timestamp to export only the closing month.
• Upload the stored file to S3, GCS, or Azure Blob by passing {{ outputs.export_index.uri }} to an upload task.
• Chain the retention delete from the OpenSearch index retention blueprint after a successful export, archive then prune.
• Reload an archive into a fresh index later with io.kestra.plugin.opensearch.Load pointed at the stored file.

Links

• OpenSearch plugin documentation
• OpenSearch scroll documentation