Elasticsearch Bulk

Elasticsearch Bulk

Certified

Replay Elasticsearch bulk file

Reads an Elasticsearch bulk file (NDJSON or ION) from Kestra storage and replays operations. Follows the bulk file contract; see Elasticsearch bulk API docs for format. ION files can be text or binary, so files written by upstream tasks are read on every Kestra version. Action-line metadata is carried onto each operation so the replay honors the file's concurrency and routing intents: routing, if_seq_no, if_primary_term, version and version_type apply to every op; pipeline and dynamic_templates apply to index and create; require_alias applies to index, create and update; retry_on_conflict applies to update. Update bodies that contain native update fields (doc, script, upsert, doc_as_upsert, scripted_upsert, detect_noop or _source) are applied with Elasticsearch bulk API semantics, so upserts only happen when the body asks for them. For compatibility, update bodies without any native update field are treated as plain partial documents with doc_as_upsert enabled, unless the action line sets if_seq_no, which Elasticsearch does not allow with upserts. A plain partial document that itself has source fields named like native update fields is ambiguous: wrap it in a doc field, or use the Load task with opType UPDATE instead.

yaml
type: io.kestra.plugin.elasticsearch.Bulk
yaml
id: elasticsearch_bulk_load
namespace: company.team

inputs:
  - id: file
    type: FILE

tasks:
  - id: bulk_load
    type: io.kestra.plugin.elasticsearch.Bulk
    connection:
      hosts:
       - "http://localhost:9200"
    from: "{{ inputs.file }}"
Properties

Elasticsearch connection

Connection settings shared by tasks; hosts are required.

Definitions
hosts*array
SubTypestring
Min items1

Elasticsearch hosts

List of HTTP(S) endpoints including scheme and port, e.g. https://elasticsearch.com: 9200; at least one is required.

basicAuth

Basic authentication

Optional HTTP basic auth credentials rendered at runtime.

passwordstring

Basic auth password

Password for HTTP basic authentication.

usernamestring

Basic auth username

Username for HTTP basic authentication.

connectTimeoutstring

Connection timeout

Maximum time to wait to establish the TCP connection to an Elasticsearch host. When unset, the Elasticsearch client default applies (1 second). A value of PT0S means no timeout.

headersarray
SubTypestring

Custom HTTP headers

Headers sent on every request in Name: Value format, e.g. Authorization: Token XYZ.

pathPrefixstring

Request path prefix

Base path prepended to every Elasticsearch endpoint, e.g. /my/path. Use only when the cluster is served behind a proxy that requires a prefix; leave empty otherwise.

responseTimeoutstring

Response timeout

Maximum time to wait for a response once the request has been sent. When unset, the Elasticsearch client default applies (unbounded), so large bulk requests are not cut off. A value of PT0S means no timeout. Set a bound (e.g. PT60S) to protect against a hung cluster.

strictDeprecationModebooleanstring

Fail on warning headers

When true, any response containing Elasticsearch warning headers is treated as an error.

targetServerVersionintegerstring
Default8

Target Elasticsearch server major version

Major version used for compatibility headers (Accept and Content-Type). Set to 8 for Elasticsearch 8 clusters or 9 for Elasticsearch 9 clusters.

trustAllSslbooleanstring

Trust all SSL certificates

WARNING — SECURITY RISK: When enabled, disables BOTH TLS certificate validation (TrustAllStrategy) AND hostname verification (NoopHostnameVerifier). This makes the connection vulnerable to man-in-the-middle attacks and must never be used in production. Prefer configuring a proper truststore for self-signed certificates instead.

Source file

Kestra internal storage URI containing bulk payload; supports dynamic rendering.

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions
assetFailureBehaviorstring
Possible Values
IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

id*string
Min length1
typestring
outputs

The assets produced as outputs.

id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*string
Min length1

Custom asset type

descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
Default1000

Bulk chunk size

Number of operations per bulk request; default 1000.

Custom shard routing

Optional routing key hashed to pick the shard instead of using the document id.

Indexed document count

Total operations sent across all bulk requests.

Unitrecords

Number of records loaded

Number of bulk requests sent

Duration of bulk requests

More Plugins in this Category

Firecrawl
Firecrawl

Tasks that scrape and crawl websites into clean, LLM-ready content with Firecrawl.

Data

2 tasks

Canonical (EE)
Canonical
Enterprise

Tasks to interact with Canonical services, enabling automation and infrastructure management through API-driven workflows.

CloudInfrastructure

5 tasks