Esql

type: "io.kestra.plugin.elasticsearch.Esql"

Query Elasticsearch using ES|QL.

This task allows you to query Elasticsearch using ES|QL.

Examples

Load data in bulk to Elasticsearch and query it using ES|QL.

id: bulk_load_and_query
namespace: company.team

tasks:
  - id: extract
    type: io.kestra.plugin.core.http.Download
    uri: https://huggingface.co/datasets/kestra/datasets/resolve/main/jsonl/books.jsonl

  - id: load
    type: io.kestra.plugin.elasticsearch.Bulk
    from: "{{ outputs.extract.uri }}"

  - id: sleep
    type: io.kestra.plugin.core.flow.Sleep
    duration: PT5S
    description: Pause needed after load before we can query

  - id: query
    type: io.kestra.plugin.elasticsearch.Esql
    fetchType: STORE
    query: |
      FROM books
        | KEEP author, name, page_count, release_date
        | SORT page_count DESC
        | LIMIT 5

pluginDefaults:
  - type: io.kestra.plugin.elasticsearch
    values:
      connection:
        headers:
          - "Authorization: ApiKey yourEncodedApiKey"
        hosts:
          - https://yourCluster.us-central1.gcp.cloud.es.io:443

Properties

connection

• Type: ElasticsearchConnection
• Dynamic: ❓
• Required: ✔️

The connection properties.

fetchType

• Type: string
• Dynamic: ❌
• Required: ✔️
• Default: FETCH
• Possible Values:
  • STORE
  • FETCH
  • FETCH_ONE
  • NONE

The way you want to store the data.

FETCH_ONE output the first row, FETCH output all the rows, STORE store all rows in a file, NONE do nothing.

query

• Type: string
• Dynamic: ✔️
• Required: ✔️

The ElasticSearch value.

Can be a JSON string. In this case, the contentType will be used or a raw Map.

filter

• Type: object
• Dynamic: ✔️
• Required: ❌

Query filter.

Specify a DSL query in the filter parameter to filter the set of documents that an ES|QL query runs on.

routing

• Type: string
• Dynamic: ✔️
• Required: ❌

Controls the shard routing of the request.

Using this value to hash the shard and not the id.

Outputs

row

• Type: object
• Required: ❌

Map containing the first row of fetched data.

Only populated if using fetchType=FETCH_ONE.

rows

• Type: array
• SubType: object
• Required: ❌

List containing the fetched data.

Only populated if using fetchType=FETCH.

size

• Type: integer
• Required: ❌

The number of fetched rows.

total

• Type: integer
• Required: ❌

The total number of rows fetched without pagination.

uri

• Type: string
• Required: ❌
• Format: uri

The URI of the data stored in Kestra's internal storage.

Only populated if using fetchType=STORE.

Definitions

io.kestra.plugin.elasticsearch.ElasticsearchConnection

Properties

hosts

• Type: array
• SubType: string
• Dynamic: ✔️
• Required: ✔️
• Min items: 1

List of HTTP ElasticSearch servers.

Must be an URI like https://elasticsearch.com:9200 with scheme and port.

basicAuth

• Type: ElasticsearchConnection-BasicAuth
• Dynamic: ❌
• Required: ❌

Basic auth configuration.

headers

• Type: array
• SubType: string
• Dynamic: ✔️
• Required: ❌

List of HTTP headers to be send on every request.

Must be a string with key value separated with :, ex: Authorization: Token XYZ.

pathPrefix

• Type: string
• Dynamic: ✔️
• Required: ❌

Sets the path's prefix for every request used by the HTTP client.

For example, if this is set to /my/path, then any client request will become /my/path/ + endpoint. In essence, every request's endpoint is prefixed by this pathPrefix. The path prefix is useful for when ElasticSearch is behind a proxy that provides a base path or a proxy that requires all paths to start with '/'; it is not intended for other purposes and it should not be supplied in other scenarios.

strictDeprecationMode

• Type: boolean
• Dynamic: ❌
• Required: ❌

Whether the REST client should return any response containing at least one warning header as a failure.

trustAllSsl

• Type: boolean
• Dynamic: ❌
• Required: ❌

Trust all SSL CA certificates.

Use this if the server is using a self signed SSL certificate.

io.kestra.plugin.elasticsearch.ElasticsearchConnection-BasicAuth

Properties

password

• Type: string
• Dynamic: ✔️
• Required: ❌

Basic auth password.

username

• Type: string
• Dynamic: ✔️
• Required: ❌

Basic auth username.