Apify Web Crawl to RAG Vector Index with Kestra

id: apify-web-data-to-rag
namespace: company.team

inputs:
  - id: start_url
    type: STRING
    displayName: URL to crawl
    description: The site Apify's Website Content Crawler will scrape for RAG ingestion.
    defaults: https://kestra.io/docs
  - id: max_items
    type: INT
    displayName: Max items
    description: Caps the crawled pages returned to control Apify cost.
    defaults: 50
  - id: question
    type: STRING
    displayName: Question
    description: A question answered from the freshly crawled content.
    defaults: What is Kestra and what are its main features?

tasks:
  - id: run_actor
    type: io.kestra.plugin.apify.actor.Run
    actorId: apify/website-content-crawler
    apiToken: "{{ secret('APIFY_API_TOKEN') }}"
    maxItems: "{{ inputs.max_items }}"
    maxTotalChargeUsd: 5
    input:
      startUrls:
        - url: "{{ inputs.start_url }}"
      maxCrawlPages: 50

  - id: get_dataset
    type: io.kestra.plugin.apify.dataset.Get
    apiToken: "{{ secret('APIFY_API_TOKEN') }}"
    datasetId: "{{ outputs.run_actor.defaultDatasetId }}"
    fields:
      - url
      - text

  - id: build_corpus
    type: io.kestra.plugin.scripts.python.Script
    inputFiles:
      dataset.json: "{{ outputs.get_dataset.dataset | json }}"
    outputFiles:
      - corpus.md
    script: |
      import json

      with open("dataset.json") as f:
          pages = json.load(f)

      with open("corpus.md", "w") as out:
          for page in pages:
              text = (page.get("text") or "").strip()
              if not text:
                  continue
              out.write("# Source: " + page.get("url", "") + "\n\n")
              out.write(text + "\n\n---\n\n")

  - id: ingest
    type: io.kestra.plugin.ai.rag.IngestDocument
    drop: true
    fromInternalURIs:
      - "{{ outputs.build_corpus.outputFiles['corpus.md'] }}"
    documentSplitter:
      splitter: RECURSIVE
      maxSegmentSizeInChars: 1000
      maxOverlapSizeInChars: 200
    provider:
      type: io.kestra.plugin.ai.provider.OpenAI
      apiKey: "{{ secret('OPENAI_API_KEY') }}"
      modelName: text-embedding-3-small
    embeddings:
      type: io.kestra.plugin.ai.embeddings.KestraKVStore

  - id: ask
    type: io.kestra.plugin.ai.agent.AIAgent
    systemMessage: |
      Answer the question using only the retrieved web content. If the answer
      is not in the retrieved context, say you could not find it. Be concise.
    prompt: "{{ inputs.question }}"
    contentRetrievers:
      - type: io.kestra.plugin.ai.retriever.EmbeddingStoreRetriever
        maxResults: 5
        minScore: 0.6
        embeddings:
          type: io.kestra.plugin.ai.embeddings.KestraKVStore
        embeddingProvider:
          type: io.kestra.plugin.ai.provider.OpenAI
          apiKey: "{{ secret('OPENAI_API_KEY') }}"
          modelName: text-embedding-3-small
    provider:
      type: io.kestra.plugin.ai.provider.OpenAI
      apiKey: "{{ secret('OPENAI_API_KEY') }}"
      modelName: gpt-4o

  - id: log_answer
    type: io.kestra.plugin.core.log.Log
    message: "{{ outputs.ask.textOutput }}"

  - id: notify
    type: io.kestra.plugin.discord.DiscordIncomingWebhook
    url: "{{ secret('DISCORD_WEBHOOK_URL') }}"
    payload: |
      {
        "content": "Apify to RAG index refreshed: {{ outputs.ingest.ingestedDocuments }} chunks embedded from {{ inputs.start_url }}. The index is ready to query."
      }

triggers:
  - id: nightly_refresh
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "0 3 * * *"

Keep a retrieval-augmented generation (RAG) knowledge base in sync with the live web. This Apify web crawling and RAG ingestion pipeline scrapes a website with Apify's Website Content Crawler, embeds the cleaned, boilerplate-free text into a vector store, and answers natural-language questions grounded in the freshly crawled content. It solves the stale-index problem that breaks RAG chatbots and AI agents: documentation, pricing, and product pages change constantly, but a one-time embedding job never catches up. Here the crawl, embed, and query steps run on a schedule so the index always reflects what is actually published.

How it works

The nightly_refresh trigger (io.kestra.plugin.core.trigger.Schedule, cron 0 3 * * *) starts the flow every night at 03:00.
run_actor (io.kestra.plugin.apify.actor.Run) runs the apify/website-content-crawler actor against start_url, capped by maxItems and maxTotalChargeUsd to control Apify cost.
get_dataset (io.kestra.plugin.apify.dataset.Get) reads the actor's defaultDatasetId and returns only the url and text fields.
build_corpus (io.kestra.plugin.scripts.python.Script) concatenates the crawled pages into a single corpus.md file, skipping empty pages.
ingest (io.kestra.plugin.ai.rag.IngestDocument) chunks the corpus with a RECURSIVE splitter and embeds it via OpenAI text-embedding-3-small into a KestraKVStore. drop: true clears the previous index first so the store mirrors the latest crawl.
ask (io.kestra.plugin.ai.agent.AIAgent) answers question using an EmbeddingStoreRetriever over the same store, with gpt-4o constrained to the retrieved context.
log_answer (io.kestra.plugin.core.log.Log) prints the answer and notify (io.kestra.plugin.discord.DiscordIncomingWebhook) posts a refresh confirmation with the chunk count.

What you get

A self-refreshing RAG index built from any public website or list of URLs.
Clean, deduplicated page text instead of raw HTML, courtesy of the Website Content Crawler.
Cost guardrails via maxItems and maxTotalChargeUsd.
A grounded question-answering step that cites only retrieved content.
A Discord notification confirming each successful refresh.

Who it's for

AI and platform engineers building RAG chatbots or agents over external content.
Data engineers who need scheduled, observable web-to-vector pipelines.
Developer relations and docs teams keeping an assistant current with live documentation.

Why orchestrate this with Kestra

Apify's scheduler can rerun a crawl, but it stops at the dataset. It cannot then chunk the output, embed it into a vector store, drop the stale index, run a grounded LLM query, and fan out a Discord alert as one observable run. Kestra wires the whole chain in declarative YAML with event and schedule triggers, per-task retries, and full execution lineage across Apify, Python, the AI plugin, and Discord. You get one auditable pipeline instead of glue scripts stitched across separate tools.

Prerequisites

A Kestra instance (local or cloud).
An Apify account with the apify/website-content-crawler actor.
An OpenAI API key with access to text-embedding-3-small and gpt-4o.
A Discord channel with an incoming webhook.

Secrets

APIFY_API_TOKEN: authenticates the Apify actor run and dataset fetch.
OPENAI_API_KEY: powers embeddings and the answering agent.
DISCORD_WEBHOOK_URL: the Discord incoming webhook for refresh notifications.

Quick start

Add APIFY_API_TOKEN, OPENAI_API_KEY, and DISCORD_WEBHOOK_URL as secrets.
Import this flow into your Kestra instance.
Run it manually, accepting the default start_url, max_items, and question, or set your own.
Check log_answer for the grounded response and your Discord channel for the confirmation.
Leave the nightly schedule on so the index stays current.

How to extend

Point start_url at your own docs, pricing, or knowledge-base site.
Swap the KestraKVStore embeddings for Pinecone, Qdrant, PGVector, or Weaviate for production scale.
Tune maxSegmentSizeInChars, maxOverlapSizeInChars, maxResults, and minScore for retrieval quality.
Replace the Discord notification with Slack, email, or a downstream subflow.
Add an event trigger so a webhook can refresh the index on demand.

Turn an Apify web crawl into a fresh RAG vector index

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Turn an Apify web crawl into a fresh RAG vector index

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More AI Blueprints

Generate and Send a Personalized AI-Powered Daily Digest Email Every Morning

AI Document Data Extraction: Parse PDFs and Web Pages into Structured JSON with OpenAI

AI Annual Report Finder and Data Strategy Summary

Summarize Text with an AI Agent, Generate Multilingual Summaries in Short, Medium, or Long Format

Generate a book chapter summary using Perplexity AI and email it.

Draft GDPR Breach Notifications for Supervisory Authorities and Data Subjects

New to Kestra?