Quickwit Ingest
CertifiedIngest documents into Quickwit
Quickwit Ingest
Ingest documents into Quickwit
Sends documents to a Quickwit index as a single NDJSON batch.
The batch is limited to 10 MB by default, so split larger datasets across several runs or use a Quickwit source instead.
type: io.kestra.plugin.quickwit.ingest.IngestExamples
Ingest a file produced by a previous task
id: quickwit_ingest
namespace: company.team
inputs:
- id: file
type: FILE
tasks:
- id: ingest
type: io.kestra.plugin.quickwit.ingest.Ingest
url: "https://quickwit.example.com:7280"
index: app-logs
from: "{{ inputs.file }}"
commit: WAIT_FOR
Ingest inline documents and check for rejections
id: quickwit_ingest_inline
namespace: company.team
tasks:
- id: ingest
type: io.kestra.plugin.quickwit.ingest.Ingest
url: "http://localhost:7280"
index: app-logs
commit: FORCE
detailedResponse: true
from:
- timestamp: "{{ now() | timestamp }}"
service: checkout
severity: ERROR
message: "Payment gateway timeout"
- timestamp: "{{ now() | timestamp }}"
service: checkout
severity: ERROR
message: "Payment gateway timeout"
- id: check
type: io.kestra.plugin.core.log.Log
message: "Ingested {{ outputs.ingest.numIngestedDocs }} / rejected {{ outputs.ingest.numRejectedDocs }}"
Properties
from *object
Documents
Documents to send, as one of:
- a single document, as a map;
- a list of documents, as a list of maps;
- a URI, either
kestra://internal storage or a localfile://path; - a JSON string holding one document or a list of documents.
Documents must match the doc mapping of the index.
index *string
Index ID
ID of the index to ingest into. The ingest API is only served by nodes running an indexer.
url *string
Quickwit URL
Base URL of the Quickwit REST API, including the scheme. The api/v1 prefix is added automatically.
A local node listens on port 7280, e.g. http://localhost: 7280.
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150basicAuth
Basic authentication
Optional HTTP basic authentication credentials.
Quickwit itself has no authentication layer, so this is only needed when the cluster sits behind a reverse proxy or an API gateway that enforces basic auth.
io.kestra.plugin.quickwit.AbstractQuickwitTask-BasicAuth
Password
Password for HTTP basic authentication.
Username
Username for HTTP basic authentication.
commit string
AUTOAUTOWAIT_FORFORCECommit behavior
When the ingested documents become visible to search: AUTO (default), WAIT_FOR or FORCE.
Use FORCE when a search runs right after this task and must already see the documents.
connectTimeout string
Connect timeout
Maximum time to wait to establish a connection to Quickwit, e.g. PT10S.
When unset, the default of the underlying HTTP client applies.
detailedResponse booleanstring
falseDetailed response
Ask Quickwit to describe every rejected document in parseFailures.
Handy to diagnose a doc mapping mismatch, but it costs performance.
headers object
Custom HTTP headers
Headers sent on every request, for example an Authorization: Bearer ... token or a gateway API key.
Quickwit itself has no authentication layer, so this is only needed when the cluster sits behind an API gateway that injects credentials.
readTimeout string
Read timeout
Maximum time to keep waiting for data on an idle connection, e.g. PT5M.
A commit=force ingest or a large search may legitimately take a while, so raise it when needed.
Outputs
numDocsForProcessing integer
Documents submitted
Number of documents submitted for processing. They may not be searchable yet.
numIngestedDocs integer
Documents ingested
Number of documents successfully persisted in the write ahead log.
numRejectedDocs integer
Documents rejected
Number of documents that could not be parsed, usually because of a doc mapping mismatch.
parseFailures array
Parse failures
Details of the rejected documents, present only when detailedResponse is enabled.
Parse failure
A document rejected by the ingest API, with the reason and the offending payload.
Rejected document
The UTF-8 decoded document chunk that generated the error.
Failure message
Human readable explanation of why the document was rejected.
Failure reason
Machine readable reason, one of invalid_json, invalid_schema or unspecified.
Metrics
records counter
recordsNumber of documents sent
