Load and search data using Elasticsearch

Build a complete extract, load, and verify pipeline for Elasticsearch with Kestra. This blueprint pulls JSON from a public REST API, indexes the document into an Elasticsearch index, and immediately validates it with both a search query and a scroll query. It is an ideal starting point for ingesting API payloads, application events, or operational records into Elasticsearch and confirming that they are queryable, without writing custom ingestion scripts.

How it works

1. The extract task (io.kestra.plugin.core.http.Request) calls the PokeAPI endpoint for the Pokemon name supplied in the pokemon input and returns the JSON response body.
2. The load task (io.kestra.plugin.elasticsearch.Put) writes that body into the local index, keyed by the Pokemon name, connecting to the cluster defined in the host variable.
3. The parallel task (io.kestra.plugin.core.flow.Parallel) runs two reads at once: search (io.kestra.plugin.elasticsearch.Search) issues a term query on the name field, and scroll (io.kestra.plugin.elasticsearch.Scroll) pages through matching documents to confirm the data is indexed and retrievable.

What you get

• A working extract, load, and verify pattern for Elasticsearch.
• A reusable Put step for indexing documents by key.
• Parallel Search and Scroll reads that prove indexing succeeded.
• A parameterized pokemon input so each run targets a different record.

Who it's for

• Data engineers loading API or event data into Elasticsearch.
• Platform teams standardizing search-index ingestion.
• Developers prototyping Elasticsearch read and write patterns.

Why orchestrate this with Kestra

Elasticsearch indexes and serves data, but it does not schedule, retry, or coordinate the upstream extraction and downstream verification around it. With Kestra you define the whole flow in declarative YAML, trigger it on a schedule or an event, add retries and alerting around each task, and capture execution lineage from the API call through indexing to the search results. The Parallel task fans the verification reads out concurrently, something an Elasticsearch query alone cannot orchestrate.

Prerequisites

• A running Elasticsearch cluster reachable at the host variable (defaults to http://host.docker.internal:9200).
• Network access from Kestra to the PokeAPI endpoint.

Secrets

This flow uses no secrets. It connects to an open local Elasticsearch instance and a public API. For a secured cluster, move credentials into Kestra secrets and reference them with {{ secret('NAME') }} in the connection block.

Quick start

1. Start a local Elasticsearch container: docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.10.2.
2. Add the flow to a namespace in your Kestra instance.
3. Execute it, optionally changing the pokemon input value.
4. Inspect the search and scroll outputs to confirm the document was indexed.

How to extend

• Swap the PokeAPI call for your own REST source or a database query.
• Point the host variable at a managed or self-hosted production cluster.
• Add a Schedule or webhook trigger to ingest on a cadence or on demand.
• Replace Put with a bulk load task for high-volume ingestion.

Links

• Plugin docs: https://kestra.io/plugins/plugin-elasticsearch