EmbeddingStoreRetriever
Embedding store content retriever for RAG (Retrieval Augmented Generation)
Embedding store content retriever for RAG (Retrieval Augmented Generation)
Embedding store content retriever for RAG (Retrieval Augmented Generation)
Retrieves relevant content from an embedding store based on semantic similarity to the query.
type: "io.kestra.plugin.ai.retriever.EmbeddingStoreRetriever"Examples
Use RAG with AIAgent using an embedding store content retriever. This example ingests documents into a KV embedding store and then uses an AI agent with the EmbeddingStoreRetriever to answer questions grounded in the ingested data.
id: "embeddingstoreretriever"
type: "io.kestra.plugin.ai.retriever.EmbeddingStoreRetriever"
id: agent_with_rag
namespace: company.ai
tasks:
- id: ingest
type: io.kestra.plugin.ai.rag.IngestDocument
provider:
type: io.kestra.plugin.ai.provider.GoogleGemini
modelName: gemini-embedding-exp-03-07
googleApiKey: "{{ kv('GEMINI_API_KEY') }}"
embeddings:
type: io.kestra.plugin.ai.embeddings.KestraKVStore
drop: true
fromDocuments:
- content: Paris is the capital of France with a population of over 2.1 million people
- content: The Eiffel Tower is the most famous landmark in Paris at 330 meters tall
- id: agent
type: io.kestra.plugin.ai.agent.AIAgent
provider:
type: io.kestra.plugin.ai.provider.GoogleGemini
modelName: gemini-2.0-flash
googleApiKey: "{{ kv('GEMINI_API_KEY') }}"
contentRetrievers:
- type: io.kestra.plugin.ai.retriever.EmbeddingStoreRetriever
embeddings:
type: io.kestra.plugin.ai.embeddings.KestraKVStore
embeddingProvider:
type: io.kestra.plugin.ai.provider.GoogleGemini
modelName: gemini-embedding-exp-03-07
googleApiKey: "{{ kv('GEMINI_API_KEY') }}"
maxResults: 3
minScore: 0.0
prompt: What is the capital of France and how many people live there?
Use multiple embedding stores simultaneously. This demonstrates the power of the content retriever approach - you can retrieve from multiple embedding stores and other sources in a single task.
id: "embeddingstoreretriever"
type: "io.kestra.plugin.ai.retriever.EmbeddingStoreRetriever"
id: multi_store_rag
namespace: company.ai
tasks:
- id: agent
type: io.kestra.plugin.ai.agent.AIAgent
provider:
type: io.kestra.plugin.ai.provider.GoogleGemini
modelName: gemini-2.0-flash
googleApiKey: "{{ kv('GEMINI_API_KEY') }}"
contentRetrievers:
- type: io.kestra.plugin.ai.retriever.EmbeddingStoreRetriever
embeddings:
type: io.kestra.plugin.ai.embeddings.Pinecone
pineconeApiKey: "{{ kv('PINECONE_API_KEY') }}"
index: technical-docs
embeddingProvider:
type: io.kestra.plugin.ai.provider.OpenAI
googleApiKey: "{{ kv('OPENAI_API_KEY') }}"
modelName: text-embedding-3-small
- type: io.kestra.plugin.ai.retriever.EmbeddingStoreRetriever
embeddings:
type: io.kestra.plugin.ai.embeddings.Qdrant
host: localhost
port: 6333
collectionName: business-docs
embeddingProvider:
type: io.kestra.plugin.ai.provider.GoogleGemini
modelName: gemini-embedding-exp-03-07
googleApiKey: "{{ kv('GEMINI_API_KEY') }}"
- type: io.kestra.plugin.ai.retriever.TavilyWebSearch
tavilyApiKey: "{{ kv('TAVILY_API_KEY') }}"
prompt: What are the latest trends in data orchestration?
Properties
embeddingProvider*RequiredNon-dynamic
Embedding model provider
Provider used to generate embeddings for the query. Must support embedding generation.
Amazon Bedrock Model Provider
AWS Access Key ID
AWS Secret Access Key
COHERECOHERETITANAmazon Bedrock Embedding Model Type
Anthropic AI Model Provider
Maximum Tokens
Specifies the maximum number of tokens that the model is allowed to generate in its response.
Azure OpenAI Model Provider
API endpoint
The Azure OpenAI endpoint in the format: https://{resource}.openai.azure.com/
Client ID
Client secret
API version
Tenant ID
DashScope (Qwen) Model Provider from Alibaba Cloud
https://dashscope-intl.aliyuncs.com/api/v1If you use a model in the China (Beijing) region, you need to replace the URL with: https://dashscope.aliyuncs.com/api/v1,
otherwise use the Singapore region of: "https://dashscope-intl.aliyuncs.com/api/v1.
The default value is computed based on the system timezone.
Whether the model uses Internet search results for reference when generating text or not
Repetition in a continuous sequence during model generation
Increasing repetition_penalty reduces the repetition in model generation,
1.0 means no penalty. Value range: (0, +inf)
Deepseek Model Provider
https://api.deepseek.com/v1GitHub Models AI Model Provider
GitHub Token
Personal Access Token (PAT) used to access GitHub Models.
Google Gemini Model Provider
Google VertexAI Model Provider
Project location
Project ID
HuggingFace Model Provider
https://router.huggingface.co/v1LocalAI Model Provider
Mistral AI Model Provider
OciGenAI Model Provider
OCID of OCI Compartment with the model
OCI Region to connect the client to
OCI SDK Authentication provider
Ollama Model Provider
Model endpoint
OpenAI Model Provider
https://api.openai.com/v1OpenRouter Model Provider
Watsonx AI Model Provider
Project Id
WorkersAI Model Provider
Account Identifier
Unique identifier assigned to an account
Base URL
Custom base URL to override the default endpoint (useful for local tests, WireMock, or enterprise gateways).
ZhiPu AI Model Provider
API Key
Model name
https://open.bigmodel.cn/API base URL
The base URL for ZhiPu API (defaults to https://open.bigmodel.cn/)
CA PEM certificate content
CA certificate as text, used to verify SSL/TLS connections when using custom endpoints.
Client PEM certificate content
PEM client certificate as text, used to authenticate the connection to enterprise AI endpoints.
The maximum retry times to request
The maximum number of tokens returned by this request
With the stop parameter, the model will automatically stop generating text when it is about to contain the specified string or token_id
embeddings*RequiredNon-dynamic
Embedding store
The embedding store to retrieve relevant content from
Chroma Embedding Store
The database base URL
Elasticsearch Embedding Store
io.kestra.plugin.ai.embeddings.Elasticsearch-ElasticsearchConnection
1List of HTTP Elasticsearch servers
Must be a URI like https://example.com: 9200 with scheme and port
Basic authorization configuration
io.kestra.plugin.ai.embeddings.Elasticsearch-ElasticsearchConnection-BasicAuth
Basic authorization password
Basic authorization username
List of HTTP headers to be sent with every request
Each item is a key: value string, e.g., Authorization: Token XYZ
Path prefix for all HTTP requests
If set to /my/path, each client request becomes /my/path/ + endpoint. Useful when Elasticsearch is behind a proxy providing a base path; do not use otherwise.
Treat responses with deprecation warnings as failures
Trust all SSL CA certificates
Use this if the server uses a self-signed SSL certificate
The name of the index to store embeddings
In-memory embedding store that stores data as Kestra KV pairs
{{flow.id}}-embedding-storeThe name of the KV pair to use
MariaDB Embedding Store
Whether to create the table if it doesn't exist
Database URL of the MariaDB database (e.g., jdbc: mariadb://host: port/dbname)
Name of the column used as the unique ID in the database
Name of the table where embeddings will be stored
Metadata Column Definitions
List of SQL column definitions for metadata fields (e.g., 'text TEXT', 'source TEXT'). Required only when using COLUMN_PER_KEY storage mode.
Metadata Index Definitions
List of SQL index definitions for metadata columns (e.g., 'INDEX idx_text (text)'). Used only with COLUMN_PER_KEY storage mode.
Metadata Storage Mode
Determines how metadata is stored: - COLUMN_PER_KEY: Use individual columns for each metadata field (requires columnDefinitions and indexes). - COMBINED_JSON (default): Store metadata as a JSON object in a single column. If columnDefinitions and indexes are provided, COLUMN_PER_KEY must be used.
Milvus Embedding Store
Token
Milvus auth token. Required if authentication is enabled; omit for local deployments without auth.
Auto flush on delete
If true, flush after delete operations.
Auto flush on insert
If true, flush after insert operations. Setting it to false can improve throughput.
Collection name
Target collection. Created automatically if it does not exist. Default: "default".
Read/write consistency level. Common values include STRONG, BOUNDED, or EVENTUALLY (depends on client/version).
Database name
Logical database to use. If not provided, the default database is used.
Milvus host name (used when uri is not set). Default: "localhost".
ID field name
Field name for document IDs. Default depends on collection schema.
Index type
Vector index type (e.g., IVF_FLAT, IVF_SQ8, HNSW). Depends on Milvus deployment and dataset.
Field name for metadata. Default depends on collection schema.
Metric type
Similarity metric (e.g., L2, IP, COSINE). Should match the embedding provider’s expected metric.
Password
Milvus port (used when uri is not set). Typical: 19530 (gRPC) or 9091 (HTTP). Default: 19530.
Retrieve embeddings on search
If true, return stored embeddings along with matches. Default: false.
Text field name
Field name for original text. Default depends on collection schema.
URI
Connection URI. Use either uri OR host/port (not both).
Examples:
- gRPC (typical): "milvus://host: 19530"
- HTTP: "http://host: 9091"
Username
Required when authentication/TLS is enabled. See https://milvus.io/docs/authenticate.md
Vector field name
Field name for the embedding vector. Must match the index definition and embedding dimensionality.
MongoDB Atlas Embedding Store
The host
The scheme (e.g., mongodb+srv)
Create the index
The database
The metadata field names
The connection string options
The password
The username
PGVector Embedding Store
The database name
The database password
The table to store embeddings in
The database user
falseWhether to use use an IVFFlat index
An IVFFlat index divides vectors into lists, and then searches a subset of those lists closest to the query vector. It has faster build times and uses less memory than HNSW but has lower query performance (in terms of speed-recall tradeoff).
Pinecone Embedding Store
The cloud provider
The index
The cloud provider region
The namespace (default will be used if not provided)
Qdrant Embedding Store
The API key
The collection name
Redis Embedding Store
The database server host
The database server port
embedding-indexThe index name
Tablestore Embedding Store
Access Key ID
The access key ID used for authentication with the database.
Access Key Secret
The access key secret used for authentication with the database.
Endpoint URL
The base URL for the Tablestore database endpoint.
Instance Name
The name of the Tablestore database instance.
Metadata Schema List
Optional list of metadata field schemas for the collection.
com.alicloud.openservices.tablestore.model.search.FieldSchema
SingleWordMaxWordMinWordSplitFuzzycom.alicloud.openservices.tablestore.model.search.analysis.AnalyzerParameter
LONGDOUBLEBOOLEANKEYWORDTEXTNESTEDGEO_POINTDATEVECTORFUZZY_KEYWORDIPJSONUNKNOWNDOCSFREQSPOSITIONSOFFSETSFLATTENNESTEDcom.alicloud.openservices.tablestore.model.search.FieldSchema
SingleWordMaxWordMinWordSplitFuzzyLONGDOUBLEBOOLEANKEYWORDTEXTNESTEDGEO_POINTDATEVECTORFUZZY_KEYWORDIPJSONUNKNOWNDOCSFREQSPOSITIONSOFFSETSFLATTENNESTEDcom.alicloud.openservices.tablestore.model.search.vector.VectorOptions
EUCLIDEANCOSINEDOT_PRODUCTWeaviate Embedding Store
API key
Weaviate API key. Omit for local deployments without auth.
Host
Cluster host name without protocol, e.g., "abc123.weaviate.network".
Avoid duplicates
If true (default), a hash-based ID is derived from each text segment to prevent duplicates. If false, a random ID is used.
ONEQUORUMALLConsistency level
Write consistency: ONE, QUORUM (default), or ALL.
gRPC port
Port for gRPC if enabled (e.g., 50051).
Metadata field name
Field used to store metadata. Defaults to "_metadata" if not set.
Metadata keys
The list of metadata keys to store - if not provided, it will default to an empty list.
Object class
Weaviate class to store objects in (must start with an uppercase letter). Defaults to "Default" if not set.
Port
Optional port (e.g., 443 for https, 80 for http). Leave unset to use provider defaults.
Scheme
Cluster scheme: "https" (recommended) or "http".
Secure gRPC
Whether the gRPC connection is secured (TLS).
Use gRPC for batch inserts
If true, use gRPC for batch inserts. HTTP remains required for search operations.
maxResultsintegerstring
3Maximum number of results to return from the embedding store
minScorenumberstring
0.0Minimum similarity score
Only results with a similarity score ≥ minScore are returned. Range: 0.0 to 1.0 inclusive.
