IngestDocument

yaml
type: "io.kestra.plugin.ai.rag.IngestDocument"

Examples

yaml
id: document_ingestion
namespace: company.ai

tasks:
  - id: ingest
    type: io.kestra.plugin.ai.rag.IngestDocument
    provider:
      type: io.kestra.plugin.ai.provider.GoogleGemini
      modelName: gemini-embedding-exp-03-07
      apiKey: "{{ kv('GEMINI_API_KEY') }}"
    embeddings:
      type: io.kestra.plugin.ai.embeddings.KestraKVStore
    drop: true
    fromExternalURLs:
      - https://raw.githubusercontent.com/kestra-io/docs/refs/heads/main/content/blogs/release-0-24.md

Properties

embeddings *

Definitions

Chroma Embedding Store

baseUrl*string

collectionName*string

type*object

Elasticsearch Embedding Store

connection*

io.kestra.plugin.ai.embeddings.Elasticsearch-ElasticsearchConnection

hosts*array

SubTypestring

Min items1

basicAuth

io.kestra.plugin.ai.embeddings.Elasticsearch-ElasticsearchConnection-BasicAuth

passwordstring

usernamestring

headersarray

SubTypestring

pathPrefixstring

strictDeprecationModebooleanstring

trustAllSslbooleanstring

indexName*string

type*object

In-memory embedding store that stores data as Kestra KV pairs

type*object

kvNamestring

Default{{flow.id}}-embedding-store

MariaDB Embedding Store

createTable*booleanstring

databaseUrl*string

fieldName*string

password*string

tableName*string

type*object

username*string

columnDefinitionsarray

SubTypestring

indexesarray

SubTypestring

metadataStorageModestring

DefaultCOLUMN_PER_KEY

Milvus Embedding Store

token*string

type*object

autoFlushOnDeletebooleanstring

autoFlushOnInsertbooleanstring

collectionNamestring

consistencyLevelstring

databaseNamestring

hoststring

idFieldNamestring

indexTypestring

metadataFieldNamestring

metricTypestring

passwordstring

portintegerstring

retrieveEmbeddingsOnSearchbooleanstring

textFieldNamestring

uristring

usernamestring

vectorFieldNamestring

MongoDB Atlas Embedding Store

collectionName*string

host*string

indexName*string

scheme*string

type*object

createIndexbooleanstring

databasestring

metadataFieldNamesarray

SubTypestring

optionsobject

passwordstring

usernamestring

PGVector Embedding Store

database*string

host*string

password*string

port*integerstring

table*string

type*object

user*string

useIndexbooleanstring

Defaultfalse

Pinecone Embedding Store

apiKey*string

cloud*string

index*string

region*string

type*object

namespacestring

Qdrant Embedding Store

apiKey*string

collectionName*string

host*string

port*integerstring

type*object

Redis Embedding Store

host*string

port*integerstring

type*object

indexNamestring

Defaultembedding-index

Tablestore Embedding Store

accessKeyId*string

accessKeySecret*string

endpoint*string

instanceName*string

type*object

metadataSchemaListarray

com.alicloud.openservices.tablestore.model.search.FieldSchema

analyzerstring

Possible Values

SingleWordMaxWordMinWordSplitFuzzy

analyzerParameter

com.alicloud.openservices.tablestore.model.search.analysis.AnalyzerParameter

dateFormatsarray

SubTypestring

enableHighlightingboolean

enableSortAndAggboolean

fieldNamestring

fieldTypestring

Possible Values

LONGDOUBLEBOOLEANKEYWORDTEXTNESTEDGEO_POINTDATEVECTORFUZZY_KEYWORDIPJSONUNKNOWN

indexboolean

indexOptionsstring

Possible Values

DOCSFREQSPOSITIONSOFFSETS

isArrayboolean

jsonTypestring

Possible Values

FLATTENNESTED

sourceFieldNamesarray

SubTypestring

storeboolean

subFieldSchemasarray

com.alicloud.openservices.tablestore.model.search.FieldSchema

analyzerstring

Possible Values

SingleWordMaxWordMinWordSplitFuzzy

analyzerParameter

dateFormatsarray

SubTypestring

enableHighlightingboolean

enableSortAndAggboolean

fieldNamestring

fieldTypestring

Possible Values

LONGDOUBLEBOOLEANKEYWORDTEXTNESTEDGEO_POINTDATEVECTORFUZZY_KEYWORDIPJSONUNKNOWN

indexboolean

indexOptionsstring

Possible Values

DOCSFREQSPOSITIONSOFFSETS

isArrayboolean

jsonTypestring

Possible Values

FLATTENNESTED

sourceFieldNamesarray

SubTypestring

storeboolean

subFieldSchemasarray

vectorOptions

com.alicloud.openservices.tablestore.model.search.vector.VectorOptions

dataTypestring

dimensioninteger

metricTypestring

Possible Values

EUCLIDEANCOSINEDOT_PRODUCT

Weaviate Embedding Store

apiKey*string

host*string

type*object

avoidDupsbooleanstring

consistencyLevelstring

Possible Values

ONEQUORUMALL

grpcPortintegerstring

metadataFieldNamestring

metadataKeysarray

SubTypestring

objectClassstring

portintegerstring

schemestring

securedGrpcbooleanstring

useGrpcForInsertsbooleanstring

provider *

Definitions

Amazon Bedrock Model Provider

accessKeyId*string

modelName*string

secretAccessKey*string

type*object

baseUrlstring

caPemstring

clientPemstring

modelTypestring

DefaultCOHERE

Possible Values

COHERETITAN

Anthropic AI Model Provider

apiKey*string

modelName*string

type*object

baseUrlstring

caPemstring

clientPemstring

maxTokensintegerstring

Azure OpenAI Model Provider

endpoint*string

modelName*string

type*object

apiKeystring

baseUrlstring

caPemstring

clientIdstring

clientPemstring

clientSecretstring

serviceVersionstring

tenantIdstring

DashScope (Qwen) Model Provider from Alibaba Cloud

apiKey*string

modelName*string

type*object

baseUrlstring

Defaulthttps://dashscope-intl.aliyuncs.com/api/v1

caPemstring

clientPemstring

enableSearchbooleanstring

maxTokensintegerstring

repetitionPenaltynumberstring

Deepseek Model Provider

apiKey*string

modelName*string

type*object

baseUrlstring

Defaulthttps://api.deepseek.com/v1

caPemstring

clientPemstring

GitHub Models AI Model Provider

gitHubToken*string

modelName*string

type*object

baseUrlstring

caPemstring

clientPemstring

Google Gemini Model Provider

apiKey*string

modelName*string

type*object

baseUrlstring

caPemstring

clientPemstring

embeddingModelConfiguration

io.kestra.plugin.ai.provider.GoogleGemini-EmbeddingModelConfiguration

maxRetriesintegerstring

outputDimensionalityintegerstring

taskTypestring

Possible Values

RETRIEVAL_QUERYRETRIEVAL_DOCUMENTSEMANTIC_SIMILARITYCLASSIFICATIONCLUSTERINGQUESTION_ANSWERINGFACT_VERIFICATION

timeoutstring

Formatduration

titleMetadataKeystring

Google VertexAI Model Provider

endpoint*string

location*string

modelName*string

project*string

type*object

baseUrlstring

caPemstring

clientPemstring

HuggingFace Model Provider

apiKey*string

modelName*string

type*object

baseUrlstring

Defaulthttps://router.huggingface.co/v1

caPemstring

clientPemstring

LocalAI Model Provider

baseUrl*string

modelName*string

type*object

caPemstring

clientPemstring

Mistral AI Model Provider

apiKey*string

modelName*string

type*object

baseUrlstring

caPemstring

clientPemstring

OciGenAI Model Provider

compartmentId*string

modelName*string

region*string

type*object

authProviderstring

baseUrlstring

caPemstring

clientPemstring

Ollama Model Provider

endpoint*string

modelName*string

type*object

baseUrlstring

caPemstring

clientPemstring

OpenAI Model Provider

apiKey*string

modelName*string

type*object

baseUrlstring

Defaulthttps://api.openai.com/v1

caPemstring

clientPemstring

OpenRouter Model Provider

apiKey*string

modelName*string

type*object

baseUrlstring

caPemstring

clientPemstring

Watsonx AI Model Provider

apiKey*string

modelName*string

projectId*string

type*object

baseUrlstring

caPemstring

clientPemstring

WorkersAI Model Provider

accountId*string

apiKey*string

modelName*string

type*object

baseUrlstring

caPemstring

clientPemstring

ZhiPu AI Model Provider

apiKey*string

modelName*string

type*object

baseUrlstring

Defaulthttps://open.bigmodel.cn/

caPemstring

clientPemstring

maxRetriesintegerstring

maxTokenintegerstring

stopsarray

SubTypestring

documentSplitter

Definitions

io.kestra.plugin.ai.rag.IngestDocument-DocumentSplitter

maxOverlapSizeInChars*integer

maxSegmentSizeInChars*integer

splitterstring

DefaultRECURSIVE

Possible Values

RECURSIVEPARAGRAPHLINESENTENCEWORD

drop booleanstring

Defaultfalse

fromDocuments array

Definitions

io.kestra.plugin.ai.rag.IngestDocument-InlineDocument

content*string

metadataobject

fromExternalURLs array

SubTypestring

fromInternalURIs array

SubTypestring

fromPath string

metadata object

SubTypestring

Outputs

embeddingStoreOutputs object

ingestedDocuments integer

inputTokenCount integer

outputTokenCount integer

totalTokenCount integer

Metrics

indexed.documents counter

Unitrecords

input.token.count counter

Unittoken

output.token.count counter

Unittoken

total.token.count counter

Unittoken

PostgreSQL

Tasks that connect to PostgreSQL via JDBC to run queries and loads.

Database

SurrealDB

Tasks that run SurrealQL queries against SurrealDB.

Database

Debezium Microsoft SQL Server

Tasks that stream Microsoft SQL Server change data capture events into Kestra with Debezium.

Database

IngestDocument

More Plugins in this Category

PostgreSQL

SurrealDB

Debezium Microsoft SQL Server

1.5.0