Nightly RAG Ingestion with Gemini Embeddings

Keep a Retrieval Augmented Generation (RAG) index fresh with Vertex AI Gemini embeddings on a Kestra schedule. This blueprint re-ingests source documents nightly, splits them into overlapping paragraph chunks, embeds each chunk with a Gemini embedding model, and writes the vectors to a store that your RAG application queries for grounded, low hallucination answers. Stale retrieval is the most common cause of bad LLM responses, and this flow solves it with a declarative, auditable pipeline.

How it works

1. The reindex_daily trigger (io.kestra.plugin.core.trigger.Schedule) fires every night at 03:00 UTC (cron 0 3 * * *), disabled by default so you can enable it after configuring secrets.
2. The ingest task (io.kestra.plugin.ai.rag.IngestDocument) pulls documents from fromExternalURLs, chunks them with a PARAGRAPH splitter (maxSegmentSizeInChars: 4096, maxOverlapSizeInChars: 200), and calls the Gemini provider (io.kestra.plugin.ai.provider.GoogleGemini, model gemini-embedding-001) to embed each chunk.
3. Vectors are written to a KestraKVStore embeddings store. drop: true rebuilds the index from scratch each run for deterministic state.
4. The notify task (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) posts a confirmation to Slack with the execution id.
5. If anything fails, the alert_on_failure error handler posts a Slack alert so the on call engineer sees it immediately.

Retries are built in: the ingest task retries up to 3 times on a 1 minute constant interval with a 15 minute timeout.

What you get

• A fresh vector index every morning, with no manual intervention.
• Deterministic rebuilds via drop: true, or incremental append by flipping the flag.
• Configurable chunking that you tune per corpus (legal, code, marketing copy).
• Slack confirmations on success and alerts on failure, with execution links for forensics.
• A foundation you extend to any vector store (Qdrant, PGVector, Pinecone, Weaviate) without rewriting the flow.

Who it's for

• AI and ML engineers building production RAG systems on Vertex AI or Gemini.
• Platform teams hosting internal chatbots over docs, wikis, and tickets.
• Data engineers who need a scheduled, observable ingestion pipeline instead of a cron script.
• Solutions architects standardizing how every team ships RAG.

Why orchestrate this with Kestra

Vertex AI gives you the embedding model, but it does not schedule, retry, version, or alert on your ingestion job. Kestra fills that gap. You get event triggers (run on a Git push, a GCS object create, or a webhook from your CMS), declarative YAML you store in Git, automatic retries with backoff, full execution lineage so you can answer "which document version is in the index right now", and namespace level secret management so API keys never live in code. When the embedding API rate limits, Kestra retries; when a document fails to fetch, you see it in the UI; when the schema changes, you diff the YAML.

Prerequisites

• A Kestra instance (open source or Enterprise Edition).
• A Google AI Studio or Vertex AI account with Gemini embedding access.
• A Slack incoming webhook URL for notifications.

Secrets

• GEMINI_API_KEY: API key for the Gemini embedding model. For production Vertex AI use, swap to a service account with IAM scoped roles.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL used by both the success notification and the error handler.

Quick start

1. Set GEMINI_API_KEY and SLACK_WEBHOOK_URL in your Kestra namespace secrets.
2. Replace the entries in fromExternalURLs with the URLs of your own documents (docs, wikis, release notes, blog posts).
3. Execute the flow manually to confirm the index is built and Slack receives the confirmation.
4. Enable the reindex_daily trigger (set disabled: false) to run on schedule.

How to extend

• Swap KestraKVStore for Qdrant, PGVector, Pinecone, or Weaviate as your corpus scales.
• Add a io.kestra.plugin.gcp.gcs.Downloads task before ingest to pull documents from a Google Cloud Storage bucket and point the splitter at the local files.
• Trigger on a Git push (io.kestra.plugin.git.Push) or a GCS object create event so the index updates the moment content lands.
• Tune maxSegmentSizeInChars and maxOverlapSizeInChars per corpus: smaller chunks for code, larger for narrative prose.
• Add a downstream RAG query task (io.kestra.plugin.ai.rag.ChatCompletion) to run smoke tests against the freshly built index before promoting it.
• Branch on outputs.ingest.numberOfSegments and fail loudly if it falls outside an expected range.

Links

• Plugin docs: https://kestra.io/plugins/plugin-ai
• Orchestration use cases: https://kestra.io/orchestration/vertex-ai