Quickwit Create
CertifiedCreate a Quickwit source
Quickwit Create
Create a Quickwit source
Attaches a streaming source, for example Kafka, Kinesis or Pulsar, to an index so Quickwit indexes it continuously.
type: io.kestra.plugin.quickwit.source.CreateExamples
Index a Kafka topic into an existing index
id: quickwit_create_source
namespace: company.team
tasks:
- id: create
type: io.kestra.plugin.quickwit.source.Create
url: "http://localhost:7280"
index: app-logs
source: kafka-source
configVersion: "0.8"
sourceType: kafka
numPipelines: 2
params:
topic: app-logs
client_params:
bootstrap.servers: "kafka:9092"
- id: report
type: io.kestra.plugin.core.log.Log
message: "Source {{ outputs.create.source.sourceId }} attached"
Properties
index *string
Index ID
ID of the index the source belongs to.
params *object
Source parameters
Parameters of the source, whose shape depends on the source type. See the source configuration documentation.
source *string
Source ID
ID of the source, unique within its index. See the source ID validation rules.
sourceType *string
Source type
Type of the source: kafka, kinesis or pulsar. It cannot be changed afterwards.
url *string
Quickwit URL
Base URL of the Quickwit REST API, including the scheme. The api/v1 prefix is added automatically.
A local node listens on port 7280, e.g. http://localhost: 7280.
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150basicAuth
Basic authentication
Optional HTTP basic authentication credentials.
Quickwit itself has no authentication layer, so this is only needed when the cluster sits behind a reverse proxy or an API gateway that enforces basic auth.
io.kestra.plugin.quickwit.AbstractQuickwitTask-BasicAuth
Password
Password for HTTP basic authentication.
Username
Username for HTTP basic authentication.
configVersion string
Configuration format version
Version of the source configuration format, which must match the version of your Quickwit
cluster, for example 0.8.
Named configVersion and not version because version is reserved by Kestra to pin a
plugin version.
connectTimeout string
Connect timeout
Maximum time to wait to establish a connection to Quickwit, e.g. PT10S.
When unset, the default of the underlying HTTP client applies.
headers object
Custom HTTP headers
Headers sent on every request, for example an Authorization: Bearer ... token or a gateway API key.
Quickwit itself has no authentication layer, so this is only needed when the cluster sits behind an API gateway that injects credentials.
numPipelines integerstring
Number of pipelines
Number of indexing pipelines running per node for this source. Quickwit defaults to 1.
readTimeout string
Read timeout
Maximum time to keep waiting for data on an idle connection, e.g. PT5M.
A commit=force ingest or a large search may legitimately take a while, so raise it when needed.
transform string
Transform
A VRL transformation applied to incoming documents,
for example .message = downcase!(.message).
Outputs
index string
Index ID
ID of the index the source was attached to.
source
Source
The source configuration as Quickwit created it.
Index source
Configuration of a source feeding a Quickwit index.
Number of pipelines
Number of indexing pipelines running per node for this source. Defaults to 1.
Source parameters
Parameters of the source, whose shape depends on the source type.
Source ID
Unique identifier of the source within its index.
Source type
Type of the source: kafka, kinesis or pulsar.
Transform
VRL transformation applied to incoming documents, if any.
Configuration format version
Format version of the source configuration, matching the Quickwit version of the cluster.
