Build a RAG Pipeline Ingest Documents and Answer Questions Using OpenAI

Build a Retrieval-Augmented Generation (RAG) pipeline that ingests documents from public URLs into a vector store, then answers natural language questions grounded in the indexed content. RAG stops large language models from hallucinating by constraining every answer to retrieved source text, so teams get accurate, citable responses over their own documentation, runbooks, and knowledge bases. This blueprint runs entirely on Kestra's native AI plugin and the Kestra KV Store, with OpenAI handling embeddings and generation, so you get a working RAG system with zero external vector database to operate.

How it works

1. ingest_documents (io.kestra.plugin.ai.rag.IngestDocument) fetches the document URLs supplied via the document_urls input, splits them with a RECURSIVE splitter (maxSegmentSizeInChars: 1000, maxOverlapSizeInChars: 200), embeds each chunk with OpenAI text-embedding-3-small, and writes vectors to a io.kestra.plugin.ai.embeddings.KestraKVStore. With drop: true, the index is rebuilt on every run to keep the knowledge base current.
2. answer_question (io.kestra.plugin.ai.agent.AIAgent) takes the question input as its prompt. Its EmbeddingStoreRetriever runs a semantic similarity search over the same KV store, returning up to maxResults: 5 chunks above minScore: 0.6. A systemMessage instructs the model to answer only from retrieved context, and gpt-4o generates the grounded response.
3. log_answer (io.kestra.plugin.core.log.Log) prints outputs.answer_question.textOutput to the run logs.

What you get

• A complete ingest-and-query RAG loop with no standalone vector database to provision.
• Chunking with overlap that preserves semantic context across segment boundaries.
• Answers constrained to source content, with an explicit fallback when nothing relevant is found.
• Configurable retrieval quality via maxResults and minScore.

Who it's for

• Data and platform engineers prototyping RAG before committing to a managed vector store.
• AI and ML teams building document Q&A over PDFs, Markdown, or web pages.
• Internal tooling teams adding search and answers over docs, runbooks, and policies.

Why orchestrate this with Kestra

A bare LLM SDK script has no scheduler, no retry semantics, and no execution history. With Kestra you can attach event triggers to re-ingest documents the moment source files change, add per-task retries to ride out transient OpenAI rate limits, and inspect full inputs, outputs, and lineage for every run from the UI. The pipeline is declarative YAML, versioned in Git, so the ingestion and retrieval configuration is reviewable and reproducible rather than buried in application code. OpenAI itself ships no orchestration layer: Kestra fills that gap with triggers, backfills, and observability around the embedding and generation calls.

Prerequisites

• A Kestra instance (local via Docker or any deployment).
• An OpenAI account with access to text-embedding-3-small and gpt-4o.
• One or more publicly reachable document URLs (PDF, TXT, HTML, Markdown).

Secrets

• OPENAI_API_KEY: OpenAI API key used for both embeddings and chat completion.

Quick start

1. Add the OPENAI_API_KEY secret to your Kestra instance.
2. Copy this blueprint into a flow.
3. Execute it with the default document_urls and question, or supply your own.
4. Open the execution and read the answer in the log_answer task output.

How to extend

• Swap KestraKVStore for a production vector store (Qdrant, Elasticsearch, Pinecone) via the embeddings configuration.
• Add a trigger that re-ingests documents on a schedule or when source files change.
• Tune maxSegmentSizeInChars, maxOverlapSizeInChars, maxResults, and minScore for your corpus.
• Chain the answer into a Slack or email notification task.

Links

• Kestra AI plugin