Nightly RAG Ingestion Pipeline with Google Gemini

Keep a Retrieval-Augmented Generation (RAG) knowledge base fresh automatically. This blueprint runs a scheduled nightly ingestion pipeline that fetches documents from external URLs, splits them into overlapping chunks, generates Google Gemini embeddings, and stores them in a vector store so a downstream chat or Q&A agent always retrieves current context. It solves the stale-index problem: when your source docs (release notes, changelogs, knowledge base articles) change, your RAG answers should change too, without anyone running an ingestion job by hand.

How it works

1. The reindex_daily trigger (io.kestra.plugin.core.trigger.Schedule) fires every night at 03:00 via the cron 0 3 * * *, before the workday starts.
2. The ingest task (io.kestra.plugin.ai.rag.IngestDocument) reads source content listed under fromExternalURLs (here, a Kestra release blog Markdown file) directly, with no intermediate file storage.
3. documentSplitter chunks each document with the PARAGRAPH splitter, a maxSegmentSizeInChars of 4096, and a maxOverlapSizeInChars of 200 so context is preserved across chunk boundaries while staying within embedding limits.
4. The provider block (io.kestra.plugin.ai.provider.GoogleGemini, model gemini-embedding-001) generates embeddings, authenticating with the GEMINI_API_KEY secret.
5. Embeddings are persisted to io.kestra.plugin.ai.embeddings.KestraKVStore, a zero-infrastructure vector store built into Kestra.
6. drop: true rebuilds the index from scratch on every run so removed or edited content never lingers as stale chunks.
7. A constant retry policy (3 attempts, 1-minute interval) plus a PT10M timeout absorb transient network or API failures without runaway executions.

What you get

• A self-updating RAG index that stays in sync with changing source documents.
• Paragraph-aware chunking with overlap for higher retrieval quality.
• A built-in vector store that needs no external database to get started.
• Built-in resilience through retries and a task timeout.

Who it's for

• AI and platform engineers building RAG assistants over documentation.
• Data teams maintaining a current semantic-search index for internal Q&A.
• Developers prototyping RAG without standing up a dedicated vector database.

Why orchestrate this with Kestra

Embedding APIs and vector stores have no native scheduler, retry logic, or run history of their own. Kestra wraps the ingestion in a declarative YAML flow with an event-driven Schedule trigger, automatic retries on transient failures, full execution lineage, and observable run logs. You get reproducible, auditable refreshes and an easy path to chain ingestion with downstream querying, alerting, or approval steps, which a standalone embedding script cannot provide.

Prerequisites

• A Kestra instance (5 minutes via Docker, or Kestra Cloud).
• A Google Gemini API key with access to the gemini-embedding-001 model.
• One or more publicly reachable source URLs to ingest.

Secrets

• GEMINI_API_KEY: your Google Gemini API key, used to authenticate embedding generation.

Quick start

1. Add GEMINI_API_KEY as a secret in your Kestra instance.
2. Replace the URL under fromExternalURLs with your own document source.
3. Add the flow and trigger one run manually to validate ingestion.
4. Confirm the schedule: subsequent runs fire automatically at 03:00 daily.
5. Point a io.kestra.plugin.ai.rag.ChatCompletion task at the same KestraKVStore embeddings to query the indexed content.

How to extend

• Swap KestraKVStore for Qdrant or PGVector as your corpus grows; the rest of the flow stays the same.
• Ingest from internal files, object storage, or a list of crawled pages instead of static URLs.
• Tune maxSegmentSizeInChars and maxOverlapSizeInChars to balance retrieval precision against cost.
• Add a downstream ChatCompletion task or a notification step that runs after ingestion completes.
• Change the cron to hourly or weekly to match how often your sources change.

Links

• plugin-ai documentation
• IngestDocument task
• ChatCompletion task