Weaviate BatchCreate
Certified
Batch insert Weaviate objects
Weaviate BatchCreate
Batch insert Weaviate objects
Loads objects into a collection over gRPC from a Kestra ION file or an inline list, in chunks of batchSize.
Each object gets a generated UUID unless uuidKey is set. If the collection is missing, Weaviate auto-schema (if enabled) creates it.
By default the task fails when any object is rejected; set failOnError: false to report rejected objects in the output instead.
After 2.0.0 the default mode is AUTO (server-side batching on Weaviate 1.36+); set mode: FIXED_SIZE for the 2.0.0 behavior of sequential batchSize requests.
type: io.kestra.plugin.weaviate.BatchCreateExamples
Load a large file in parallel requests of fixed size; AUTO (the default) would use server-side batching on Weaviate 1.36+.
id: weaviate_batch_parallel
namespace: company.team
inputs:
- id: file
type: FILE
tasks:
- id: load
type: io.kestra.plugin.weaviate.BatchCreate
url: http://localhost:8080
className: Product
objects: "{{ inputs.file }}"
mode: FIXED_SIZE
batchSize: 200
concurrentRequests: 4
Insert an inline list of objects.
id: weaviate_batch_load
namespace: company.team
tasks:
- id: batch_load
type: io.kestra.plugin.weaviate.BatchCreate
url: https://demo-cluster-id.weaviate.cloud
apiKey: "{{ secret('WEAVIATE_API_KEY') }}"
className: WeaviateDemo
objects:
- textField: "some text"
numField: 24
- textField: "another text"
numField: 42
Insert objects from an ION file produced by another task.
id: weaviate_batch_insert
namespace: company.team
tasks:
- id: extract
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/ion/ingest.ion
- id: batch_insert
type: io.kestra.plugin.weaviate.BatchCreate
url: https://demo-cluster-id.weaviate.cloud
apiKey: "{{ secret('WEAVIATE_API_KEY') }}"
className: Titles
objects: "{{ outputs.extract.uri }}"
Insert objects with their own IDs and precomputed vectors, and keep going if some objects are rejected.
id: weaviate_batch_vectors
namespace: company.team
tasks:
- id: batch_insert
type: io.kestra.plugin.weaviate.BatchCreate
url: http://localhost:8080
className: Documents
uuidKey: id
vectorKey: embedding
failOnError: false
objects:
- id: 0b9e8a4a-3b1c-4c1e-9f7a-1c2d3e4f5a6b
title: First document
embedding: [0.12, 0.45, 0.91]
Installation
Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command:
kestractl plugins get io.kestra.plugin:plugin-weaviate --compatible-for $KESTRA_VERSION For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.
Properties
className *string
Collection (class) to insert into
objects *stringarray
Objects to insert with properties
Either a Kestra storage URI to an ION file or an inline list of maps. Each entry becomes one Weaviate object.
url *string
1Configure Weaviate endpoint URL
HTTP(S) URL of the Weaviate server, including the port when it is not the scheme default. Without a scheme, https is assumed. Example: http://localhost: 8080 or https://cluster-id.weaviate.cloud. Weaviate Cloud URLs (*.weaviate.cloud, *.weaviate.network) are detected automatically. Requires Weaviate 1.32 or later.
apiKey string
Provide managed cluster API key
Optional API key; required for Weaviate Cloud. If omitted, requests use anonymous access.
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150batchSize integerstring
500Number of objects sent per request
Between 1 and 10000.
concurrentRequests integerstring
1Requests sent in parallel (FIXED_SIZE)
Between 1 and 32.
failOnError booleanstring
trueFail when an object is rejected
When false, rejected objects are listed in the output errors (up to 100) and the task succeeds.
grpcHost string
Override gRPC host
Host of the Weaviate gRPC endpoint, used for queries and batch inserts.
Defaults to the host of url, or to the grpc- prefixed host on Weaviate Cloud.
grpcPort integerstring
Override gRPC port
Port of the Weaviate gRPC endpoint, between 1 and 65535. Defaults to 50051, or 443 on Weaviate Cloud.
gRPC uses TLS when url is https.
headers object
Set custom request headers
Key/value headers sent with every REST and gRPC call, e.g. vectorizer API keys such as X-OpenAI-Api-Key.
Values that are credentials should come from {{ secret('...') }}.
The plugin also sends an X-Weaviate-Client-Integration header identifying itself; setting that key here overrides it.
mode string
AUTOAUTOSERVER_SIDEDYNAMICFIXED_SIZERATE_LIMITHow objects are sent
AUTO (default) uses SERVER_SIDE on Weaviate 1.36 or later and DYNAMIC on older servers.
SERVER_SIDE streams objects and lets Weaviate set the pace from its load; batchSize is only the initial size.
DYNAMIC adapts the batch size to how long each request takes, starting from batchSize.
FIXED_SIZE sends batchSize objects per request, concurrentRequests at a time.
RATE_LIMIT sends at most objectsPerMinute, e.g. to stay under a vectorizer provider's rate limit.
Changed after 2.0.0: the default used to be sequential requests of batchSize objects; set mode: FIXED_SIZE to keep that behavior.
See https://docs.weaviate.io/weaviate/manage-objects/import.
objectsPerMinute integerstring
Maximum objects sent per minute (RATE_LIMIT)
Required with RATE_LIMIT.
requestTimeout string
Set request timeout
Timeout applied to connection, query and insert requests. Defaults to the Weaviate client defaults (30s connect, 60s query, 120s insert).
targetVector string
Name of the vector vectorKey fills
For collections with named vectors. Defaults to the default vector.
tenant string
Tenant
Tenant to insert into, for multi-tenant collections.
uuidKey string
Key holding the object UUID
When set, this key is removed from each object and used as its UUID; objects without it get a generated UUID. Inserting an existing UUID replaces that object.
vectorKey string
Key holding the object vector
When set, this key is removed from each object and used as its vector (a list of numbers, or a list of lists for multi-vectors). Use this for collections without a vectorizer.
Outputs
errors array
Rejected objects
Up to 100 entries, in input order.
io.kestra.plugin.weaviate.BatchCreate-ObjectError
Zero-based position of the object in the input
Error returned by Weaviate
UUID of the rejected object
failedCount integer
0Number of objects rejected
insertedCount integer
0Number of objects inserted
mode string
AUTOSERVER_SIDEDYNAMICFIXED_SIZERATE_LIMITBatching mode used
With AUTO, the mode it resolved to: SERVER_SIDE or DYNAMIC.
requestCount integer
Insert requests sent
Not reported for SERVER_SIDE, where Weaviate sets the batches.