Weaviate BatchCreate

Weaviate BatchCreate

Certified

Batch insert Weaviate objects

Loads objects into a collection over gRPC from a Kestra ION file or an inline list, in chunks of batchSize. Each object gets a generated UUID unless uuidKey is set. If the collection is missing, Weaviate auto-schema (if enabled) creates it. By default the task fails when any object is rejected; set failOnError: false to report rejected objects in the output instead. After 2.0.0 the default mode is AUTO (server-side batching on Weaviate 1.36+); set mode: FIXED_SIZE for the 2.0.0 behavior of sequential batchSize requests.

yaml
type: io.kestra.plugin.weaviate.BatchCreate

Load a large file in parallel requests of fixed size; AUTO (the default) would use server-side batching on Weaviate 1.36+.

yaml
id: weaviate_batch_parallel
namespace: company.team

inputs:
  - id: file
    type: FILE

tasks:
  - id: load
    type: io.kestra.plugin.weaviate.BatchCreate
    url: http://localhost:8080
    className: Product
    objects: "{{ inputs.file }}"
    mode: FIXED_SIZE
    batchSize: 200
    concurrentRequests: 4

Insert an inline list of objects.

yaml
id: weaviate_batch_load
namespace: company.team

tasks:
  - id: batch_load
    type: io.kestra.plugin.weaviate.BatchCreate
    url: https://demo-cluster-id.weaviate.cloud
    apiKey: "{{ secret('WEAVIATE_API_KEY') }}"
    className: WeaviateDemo
    objects:
      - textField: "some text"
        numField: 24
      - textField: "another text"
        numField: 42

Insert objects from an ION file produced by another task.

yaml
id: weaviate_batch_insert
namespace: company.team

tasks:
  - id: extract
    type: io.kestra.plugin.core.http.Download
    uri: https://huggingface.co/datasets/kestra/datasets/raw/main/ion/ingest.ion

  - id: batch_insert
    type: io.kestra.plugin.weaviate.BatchCreate
    url: https://demo-cluster-id.weaviate.cloud
    apiKey: "{{ secret('WEAVIATE_API_KEY') }}"
    className: Titles
    objects: "{{ outputs.extract.uri }}"

Insert objects with their own IDs and precomputed vectors, and keep going if some objects are rejected.

yaml
id: weaviate_batch_vectors
namespace: company.team

tasks:
  - id: batch_insert
    type: io.kestra.plugin.weaviate.BatchCreate
    url: http://localhost:8080
    className: Documents
    uuidKey: id
    vectorKey: embedding
    failOnError: false
    objects:
      - id: 0b9e8a4a-3b1c-4c1e-9f7a-1c2d3e4f5a6b
        title: First document
        embedding: [0.12, 0.45, 0.91]
Installation

Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command:

kestractl plugins get io.kestra.plugin:plugin-weaviate --compatible-for $KESTRA_VERSION

For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.

Properties

Collection (class) to insert into

Objects to insert with properties

Either a Kestra storage URI to an ION file or an inline list of maps. Each entry becomes one Weaviate object.

Min length1

Configure Weaviate endpoint URL

HTTP(S) URL of the Weaviate server, including the port when it is not the scheme default. Without a scheme, https is assumed. Example: http://localhost: 8080 or https://cluster-id.weaviate.cloud. Weaviate Cloud URLs (*.weaviate.cloud, *.weaviate.network) are detected automatically. Requires Weaviate 1.32 or later.

Provide managed cluster API key

Optional API key; required for Weaviate Cloud. If omitted, requests use anonymous access.

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions
assetFailureBehaviorstring
Possible Values
IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

id*string
Min length1
typestring
outputs

The assets produced as outputs.

id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*string
Min length1

Custom asset type

descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
Default500

Number of objects sent per request

Between 1 and 10000.

Default1

Requests sent in parallel (FIXED_SIZE)

Between 1 and 32.

Defaulttrue

Fail when an object is rejected

When false, rejected objects are listed in the output errors (up to 100) and the task succeeds.

Override gRPC host

Host of the Weaviate gRPC endpoint, used for queries and batch inserts. Defaults to the host of url, or to the grpc- prefixed host on Weaviate Cloud.

Override gRPC port

Port of the Weaviate gRPC endpoint, between 1 and 65535. Defaults to 50051, or 443 on Weaviate Cloud. gRPC uses TLS when url is https.

Set custom request headers

Key/value headers sent with every REST and gRPC call, e.g. vectorizer API keys such as X-OpenAI-Api-Key. Values that are credentials should come from {{ secret('...') }}. The plugin also sends an X-Weaviate-Client-Integration header identifying itself; setting that key here overrides it.

DefaultAUTO
Possible Values
AUTOSERVER_SIDEDYNAMICFIXED_SIZERATE_LIMIT

How objects are sent

AUTO (default) uses SERVER_SIDE on Weaviate 1.36 or later and DYNAMIC on older servers. SERVER_SIDE streams objects and lets Weaviate set the pace from its load; batchSize is only the initial size. DYNAMIC adapts the batch size to how long each request takes, starting from batchSize. FIXED_SIZE sends batchSize objects per request, concurrentRequests at a time. RATE_LIMIT sends at most objectsPerMinute, e.g. to stay under a vectorizer provider's rate limit. Changed after 2.0.0: the default used to be sequential requests of batchSize objects; set mode: FIXED_SIZE to keep that behavior. See https://docs.weaviate.io/weaviate/manage-objects/import.

Maximum objects sent per minute (RATE_LIMIT)

Required with RATE_LIMIT.

Set request timeout

Timeout applied to connection, query and insert requests. Defaults to the Weaviate client defaults (30s connect, 60s query, 120s insert).

Name of the vector vectorKey fills

For collections with named vectors. Defaults to the default vector.

Tenant

Tenant to insert into, for multi-tenant collections.

Key holding the object UUID

When set, this key is removed from each object and used as its UUID; objects without it get a generated UUID. Inserting an existing UUID replaces that object.

Key holding the object vector

When set, this key is removed from each object and used as its vector (a list of numbers, or a list of lists for multi-vectors). Use this for collections without a vectorizer.

Rejected objects

Up to 100 entries, in input order.

Definitions
indexinteger

Zero-based position of the object in the input

messagestring

Error returned by Weaviate

uuidstring

UUID of the rejected object

Default0

Number of objects rejected

Default0

Number of objects inserted

Possible Values
AUTOSERVER_SIDEDYNAMICFIXED_SIZERATE_LIMIT

Batching mode used

With AUTO, the mode it resolved to: SERVER_SIDE or DYNAMIC.

Insert requests sent

Not reported for SERVER_SIDE, where Weaviate sets the batches.