Extract data from a REST API, load it to Weaviate and query it with GraphQL

Build a fully orchestrated ingestion and retrieval pipeline for the Weaviate vector database. This blueprint downloads a JSON dataset from a public HTTP endpoint, converts it to Kestra's internal Ion format, batch loads it into a Weaviate class, and then runs GraphQL queries against the cluster, including an optional generative search step powered by OpenAI. It solves the common gap between data sources and vector stores: most teams need a reliable way to ingest documents into Weaviate on a schedule or in response to upstream events, without writing custom Python glue.

How it works

1. io.kestra.plugin.core.http.Download (json) pulls a sample Jeopardy dataset from a public GitHub raw URL.
2. io.kestra.plugin.serdes.json.JsonToIon (json_to_ion) converts the JSON payload into Kestra's Ion format so it can be streamed efficiently to downstream tasks.
3. io.kestra.plugin.weaviate.BatchCreate (batch_load) ingests the Ion file into the Questions class on Weaviate, using an internal storage URI.
4. A second io.kestra.plugin.weaviate.BatchCreate (batch_load_map) shows the inline key-value pattern by loading a small Users collection from literal YAML maps.
5. io.kestra.plugin.weaviate.Query (query_users) runs a GraphQL Get query that returns the first ten Questions with answer, category, and question fields.
6. A disabled io.kestra.plugin.weaviate.Query (generative_search) demonstrates nearText semantic search wired to OpenAI through the X-OpenAI-Api-Key header.

What you get

• Two complementary ingestion patterns: file-based (objects from a storage URI) and inline (objects as a list of maps).
• A working GraphQL query example you can adapt to your own classes.
• A starting point for generative and hybrid search using nearText and external model providers.
• Clean separation of credentials from code via Kestra secrets.

Who it's for

• AI and ML engineers building RAG pipelines on top of Weaviate.
• Data engineers who need to keep a vector index in sync with upstream sources.
• Platform teams standardizing vector database ingestion across multiple projects.

Why orchestrate this with Kestra

Weaviate itself has no native scheduler, no retry semantics, and no built-in way to coordinate ingestion with upstream extraction. Kestra adds event triggers (S3, GCS, Kafka, webhooks), automatic retries with backoff, full execution lineage, and declarative YAML that you can version in Git. You get observability into every batch load, the ability to fan out across many classes in parallel, and a single control plane that ties Weaviate to the rest of your data stack.

Prerequisites

• A running Weaviate cluster (Weaviate Cloud or self-hosted). The blueprint points at a demo URL, replace it with your own.
• Network access from your Kestra worker to the Weaviate endpoint.
• Optional: an OpenAI API key if you want to enable the generative search task.

Secrets

• WEAVIATE_API_KEY: API key for your Weaviate cluster.
• OPENAI_API_KEY: only required if you enable the generative_search task.

Quick start

1. Create a Weaviate cluster and generate an API key.
2. Store the key as the WEAVIATE_API_KEY secret in Kestra.
3. Replace the demo url in every Weaviate task with your own cluster endpoint.
4. Execute the flow and inspect the query_users output for the returned Questions.
5. (Optional) Add OPENAI_API_KEY, remove disabled: true on generative_search, and rerun for semantic search.

How to extend

• Replace the HTTP Download task with an extractor from your real source: Postgres, Snowflake, S3, Kafka, or a SaaS API.
• Add a schema bootstrap step that calls Weaviate's REST API to create classes with custom vectorizers before batch_load runs.
• Fan out ingestion across multiple classes using ForEach over a list of source files.
• Wire a Schedule or event trigger (for example io.kestra.plugin.aws.s3.Trigger) to refresh the vector index whenever new documents land.
• Chain the GraphQL result into a downstream task that posts answers to Slack or writes them back to a database.

Links

• Weaviate plugin docs: https://kestra.io/plugins/plugin-weaviate