IngestDocument
Currently supports text documents (TXT, HTML, Markdown).
Currently supports text documents (TXT, HTML, Markdown).
Ingest documents into an embedding store
Currently supports text documents (TXT, HTML, Markdown).
type: "io.kestra.plugin.ai.rag.IngestDocument"Examples
Ingest documents into a KV embedding store. WARNING: the KV embedding store is for quick prototyping only; it stores embedding vectors in a KV store and loads them all into memory.
id: document_ingestion
namespace: company.ai
tasks:
- id: ingest
type: io.kestra.plugin.ai.rag.IngestDocument
provider:
type: io.kestra.plugin.ai.provider.GoogleGemini
modelName: gemini-embedding-exp-03-07
apiKey: "{{ kv('GEMINI_API_KEY') }}"
embeddings:
type: io.kestra.plugin.ai.embeddings.KestraKVStore
drop: true
fromExternalURLs:
- https://raw.githubusercontent.com/kestra-io/docs/refs/heads/main/content/blogs/release-0-24.md
Properties
embeddings*RequiredNon-dynamic
Embedding store provider
Chroma Embedding Store
The database base URL
Elasticsearch Embedding Store
io.kestra.plugin.ai.embeddings.Elasticsearch-ElasticsearchConnection
1List of HTTP Elasticsearch servers
Must be a URI like https://example.com: 9200 with scheme and port
Basic authorization configuration
io.kestra.plugin.ai.embeddings.Elasticsearch-ElasticsearchConnection-BasicAuth
Basic authorization password
Basic authorization username
List of HTTP headers to be sent with every request
Each item is a key: value string, e.g., Authorization: Token XYZ
Path prefix for all HTTP requests
If set to /my/path, each client request becomes /my/path/ + endpoint. Useful when Elasticsearch is behind a proxy providing a base path; do not use otherwise.
Treat responses with deprecation warnings as failures
Trust all SSL CA certificates
Use this if the server uses a self-signed SSL certificate
The name of the index to store embeddings
In-memory embedding store that stores data as Kestra KV pairs
{{flow.id}}-embedding-storeThe name of the KV pair to use
MariaDB Embedding Store
Whether to create the table if it doesn't exist
Database URL of the MariaDB database (e.g., jdbc: mariadb://host: port/dbname)
Name of the column used as the unique ID in the database
Name of the table where embeddings will be stored
Metadata Column Definitions
List of SQL column definitions for metadata fields (e.g., 'text TEXT', 'source TEXT'). Required only when using COLUMN_PER_KEY storage mode.
Metadata Index Definitions
List of SQL index definitions for metadata columns (e.g., 'INDEX idx_text (text)'). Used only with COLUMN_PER_KEY storage mode.
Metadata Storage Mode
Determines how metadata is stored: - COLUMN_PER_KEY: Use individual columns for each metadata field (requires columnDefinitions and indexes). - COMBINED_JSON (default): Store metadata as a JSON object in a single column. If columnDefinitions and indexes are provided, COLUMN_PER_KEY must be used.
Milvus Embedding Store
Token
Milvus auth token. Required if authentication is enabled; omit for local deployments without auth.
Auto flush on delete
If true, flush after delete operations.
Auto flush on insert
If true, flush after insert operations. Setting it to false can improve throughput.
Collection name
Target collection. Created automatically if it does not exist. Default: "default".
Read/write consistency level. Common values include STRONG, BOUNDED, or EVENTUALLY (depends on client/version).
Database name
Logical database to use. If not provided, the default database is used.
Milvus host name (used when uri is not set). Default: "localhost".
ID field name
Field name for document IDs. Default depends on collection schema.
Index type
Vector index type (e.g., IVF_FLAT, IVF_SQ8, HNSW). Depends on Milvus deployment and dataset.
Field name for metadata. Default depends on collection schema.
Metric type
Similarity metric (e.g., L2, IP, COSINE). Should match the embedding provider’s expected metric.
Password
Milvus port (used when uri is not set). Typical: 19530 (gRPC) or 9091 (HTTP). Default: 19530.
Retrieve embeddings on search
If true, return stored embeddings along with matches. Default: false.
Text field name
Field name for original text. Default depends on collection schema.
URI
Connection URI. Use either uri OR host/port (not both).
Examples:
- gRPC (typical): "milvus://host: 19530"
- HTTP: "http://host: 9091"
Username
Required when authentication/TLS is enabled. See https://milvus.io/docs/authenticate.md
Vector field name
Field name for the embedding vector. Must match the index definition and embedding dimensionality.
MongoDB Atlas Embedding Store
The host
The scheme (e.g., mongodb+srv)
Create the index
The database
The metadata field names
The connection string options
The password
The username
PGVector Embedding Store
The database name
The database password
The table to store embeddings in
The database user
falseWhether to use use an IVFFlat index
An IVFFlat index divides vectors into lists, and then searches a subset of those lists closest to the query vector. It has faster build times and uses less memory than HNSW but has lower query performance (in terms of speed-recall tradeoff).
Pinecone Embedding Store
The cloud provider
The index
The cloud provider region
The namespace (default will be used if not provided)
Qdrant Embedding Store
The API key
The collection name
Redis Embedding Store
The database server host
The database server port
embedding-indexThe index name
Tablestore Embedding Store
Access Key ID
The access key ID used for authentication with the database.
Access Key Secret
The access key secret used for authentication with the database.
The base URL for the Tablestore database endpoint.
Instance Name
The name of the Tablestore database instance.
Metadata Schema List
Optional list of metadata field schemas for the collection.
com.alicloud.openservices.tablestore.model.search.FieldSchema
SingleWordMaxWordMinWordSplitFuzzycom.alicloud.openservices.tablestore.model.search.analysis.AnalyzerParameter
LONGDOUBLEBOOLEANKEYWORDTEXTNESTEDGEO_POINTDATEVECTORFUZZY_KEYWORDIPJSONUNKNOWNDOCSFREQSPOSITIONSOFFSETSFLATTENNESTEDcom.alicloud.openservices.tablestore.model.search.FieldSchema
SingleWordMaxWordMinWordSplitFuzzyLONGDOUBLEBOOLEANKEYWORDTEXTNESTEDGEO_POINTDATEVECTORFUZZY_KEYWORDIPJSONUNKNOWNDOCSFREQSPOSITIONSOFFSETSFLATTENNESTEDcom.alicloud.openservices.tablestore.model.search.vector.VectorOptions
EUCLIDEANCOSINEDOT_PRODUCTWeaviate Embedding Store
API key
Weaviate API key. Omit for local deployments without auth.
Host
Cluster host name without protocol, e.g., "abc123.weaviate.network".
Avoid duplicates
If true (default), a hash-based ID is derived from each text segment to prevent duplicates. If false, a random ID is used.
ONEQUORUMALLConsistency level
Write consistency: ONE, QUORUM (default), or ALL.
gRPC port
Port for gRPC if enabled (e.g., 50051).
Metadata field name
Field used to store metadata. Defaults to "_metadata" if not set.
Metadata keys
The list of metadata keys to store - if not provided, it will default to an empty list.
Object class
Weaviate class to store objects in (must start with an uppercase letter). Defaults to "Default" if not set.
Port
Optional port (e.g., 443 for https, 80 for http). Leave unset to use provider defaults.
Scheme
Cluster scheme: "https" (recommended) or "http".
Secure gRPC
Whether the gRPC connection is secured (TLS).
Use gRPC for batch inserts
If true, use gRPC for batch inserts. HTTP remains required for search operations.
provider*RequiredNon-dynamic
Language model provider
Must be configured with an embedding model.
Amazon Bedrock Model Provider
AWS Access Key ID
AWS Secret Access Key
COHERECOHERETITANAmazon Bedrock Embedding Model Type
Anthropic AI Model Provider
Maximum Tokens
Specifies the maximum number of tokens that the model is allowed to generate in its response.
Azure OpenAI Model Provider
API endpoint
The Azure OpenAI endpoint in the format: https://{resource}.openai.azure.com/
Client ID
Client secret
API version
Tenant ID
DashScope (Qwen) Model Provider from Alibaba Cloud
https://dashscope-intl.aliyuncs.com/api/v1If you use a model in the China (Beijing) region, you need to replace the URL with: https://dashscope.aliyuncs.com/api/v1,
otherwise use the Singapore region of: "https://dashscope-intl.aliyuncs.com/api/v1.
The default value is computed based on the system timezone.
Whether the model uses Internet search results for reference when generating text or not
Repetition in a continuous sequence during model generation
Increasing repetition_penalty reduces the repetition in model generation,
1.0 means no penalty. Value range: (0, +inf)
Deepseek Model Provider
https://api.deepseek.com/v1GitHub Models AI Model Provider
GitHub Token
Personal Access Token (PAT) used to access GitHub Models.
Google Gemini Model Provider
Google VertexAI Model Provider
Endpoint URL
Project location
Project ID
HuggingFace Model Provider
https://router.huggingface.co/v1LocalAI Model Provider
Mistral AI Model Provider
OciGenAI Model Provider
OCID of OCI Compartment with the model
OCI Region to connect the client to
OCI SDK Authentication provider
Ollama Model Provider
Model endpoint
OpenAI Model Provider
https://api.openai.com/v1OpenRouter Model Provider
WorkersAI Model Provider
Account Identifier
Unique identifier assigned to an account
Base URL
Custom base URL to override the default endpoint (useful for local tests, WireMock, or enterprise gateways).
ZhiPu AI Model Provider
API Key
Model name
https://open.bigmodel.cn/API base URL
The base URL for ZhiPu API (defaults to https://open.bigmodel.cn/)
CA PEM certificate content
CA certificate as text, used to verify SSL/TLS connections when using custom endpoints.
Client PEM certificate content
PEM client certificate as text, used to authenticate the connection to enterprise AI endpoints.
The maximum retry times to request
The maximum number of tokens returned by this request
With the stop parameter, the model will automatically stop generating text when it is about to contain the specified string or token_id
documentSplitterNon-dynamic
Document splitter
io.kestra.plugin.ai.rag.IngestDocument-DocumentSplitter
Maximum overlap size (characters). Only full sentences are considered for overlap.
Maximum segment size (characters)
RECURSIVERECURSIVEPARAGRAPHLINESENTENCEWORDDocumentSplitter type
Recommended: RECURSIVE for generic text. It splits into paragraphs first and fits as many as possible into a single TextSegment. If paragraphs are too long, they are recursively split into lines, then sentences, then words, then characters until they fit into a segment.
dropbooleanstring
falseDrop the store before ingestion (useful for testing)
fromDocumentsNon-dynamicarray
List of inline documents
io.kestra.plugin.ai.rag.IngestDocument-InlineDocument
Document content
Document metadata
fromExternalURLsarray
List of document URLs from external sources
fromInternalURIsarray
List of internal storage URIs for documents
Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.
fromPathstring
Path in the task working directory containing documents to ingest
Each document in the directory will be ingested into the embedding store. Ingestion is recursive and protected against path traversal (CWE-22).
metadataobject
Additional metadata to add to all ingested documents
Outputs
embeddingStoreOutputsobject
Additional outputs from the embedding store
ingestedDocumentsinteger
Number of ingested documents
inputTokenCountinteger
Input token count
outputTokenCountinteger
Output token count
totalTokenCountinteger
Total token count
Metrics
indexed.documentscounter
recordsNumber of indexed documents
input.token.countcounter
tokenLarge Language Model (LLM) input token count
output.token.countcounter
tokenLarge Language Model (LLM) output token count
total.token.countcounter
tokenLarge Language Model (LLM) total token count
