LanceDB Upsert

LanceDB Upsert

Certified

Upsert vector records into a LanceDB table

Inserts or updates vector records and metadata into a LanceDB table. Accepts inline record lists or Kestra internal storage files.

yaml
type: io.kestra.plugin.lancedb.Upsert

Upsert embeddings from an upstream task output file

yaml
id: lancedb_upsert
namespace: company.team

tasks:
  - id: upsert
    type: io.kestra.plugin.lancedb.Upsert
    uri: "s3://my-bucket/lancedb"
    table: documents
    from: "{{ outputs.embed.uri }}"
    storageOptions:
      aws_access_key_id: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      aws_secret_access_key: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
      region: "us-east-1"

Upsert inline vector records with metadata

yaml
id: lancedb_upsert_inline
namespace: company.team

tasks:
  - id: upsert
    type: io.kestra.plugin.lancedb.Upsert
    uri: "/tmp/lancedb"
    table: documents
    batchSize: 100
    from:
      - id: "doc1"
        vector: [0.05, 0.61, 0.76, 0.74]
        category: "docs"
        title: "Getting Started"
      - id: "doc2"
        vector: [0.19, 0.81, 0.75, 0.11]
        category: "api"
        title: "API Reference"
Properties

Records to upsert

Data source, either an inline list of maps or a Kestra storage URI to an ION/JSON file. Each record should contain a 'vector' list of numbers and optional metadata fields.

Table name

The name of the LanceDB table to upsert records into.

Database URI

Path or URI to the LanceDB database directory or endpoint, e.g. local directory '/tmp/lancedb', cloud object store 's3://my-bucket/lancedb', 'gs://my-bucket/lancedb', 'az://my-bucket/lancedb'.

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions
assetFailureBehaviorstring
Possible Values
IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

id*string
Min length1
typestring
outputs

The assets produced as outputs.

id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*string
Min length1

Custom asset type

descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
Default1000

Batch size

Number of records written per batch, between 1 and 100000 (default: 1000).

Storage options

Key-value configuration options for cloud object storage (e.g. AWS access key, secret key, region, endpoint URL).

Default0

Record count

The total count of records successfully upserted into the table.

More Plugins in this Category

Ashby
Ashby

Ashby plugin for Kestra

Data

1 tasks

1 blueprints

Dremio
Dremio

Tasks that connect to Dremio via JDBC to run queries and loads.

Data

3 tasks

3 blueprints