Extract data from a REST API, load it to Weaviate and query it with GraphQL

Source

yaml

id: weaviate-load-and-query
namespace: company.team

tasks:
  - id: json
    type: io.kestra.plugin.core.http.Download
    uri: https://raw.githubusercontent.com/weaviate-tutorials/quickstart/main/data/jeopardy_tiny.json

  - id: json_to_ion
    type: io.kestra.plugin.serdes.json.JsonToIon
    from: "{{ outputs.json.uri }}"
    newLine: false

  - id: batch_load
    type: io.kestra.plugin.weaviate.BatchCreate
    url: https://demo-oczq9ryw.weaviate.network
    apiKey: "{{ secret('WEAVIATE_API_KEY') }}"
    className: Questions
    objects: "{{ outputs.json_to_ion.uri }}"

  - id: batch_load_map
    type: io.kestra.plugin.weaviate.BatchCreate
    url: demo-oczq9ryw.weaviate.network
    apiKey: "{{ secret('WEAVIATE_API_KEY') }}"
    className: Users
    objects:
      - company: kestra
        user: Anna
        city: Berlin
      - company: initech
        user: Peter Gibbons
        city: Austin
      - company: initech
        user: Bill Lumbergh
        city: Austin
      - company: initech
        user: Bob Slydell
        city: Austin

  - id: query_users
    type: io.kestra.plugin.weaviate.Query
    url: demo-oczq9ryw.weaviate.network
    apiKey: "{{ secret('WEAVIATE_API_KEY') }}"
    query: |
      {
        Get {
          Questions(limit: 10) {
            answer
            category
            question
          }
        }
      }      

  - id: generative_search
    type: io.kestra.plugin.weaviate.Query
    disabled: true
    url: https://demo-oczq9ryw.weaviate.network
    apiKey: "{{ secret('WEAVIATE_API_KEY') }}"
    headers:
      X-OpenAI-Api-Key: "{{ secret('OPENAI_API_KEY') }}"
    query: |
      {
        Get {
          Question(limit: 5, nearText: {concepts: ["biology"]}) {
            answer
            category
            question
          }
        }
      }

About this blueprint

Ingest AI API

This flow shows how to extract data from an HTTP API, load it to a Weaviate cluster and query it with GraphQL. This flow assumes that you have a Weaviate cluster running, and that you created an API key. Make sure to replace the url and apiKey values in the tasks with your Weaviate credentials. It's recommended to use Secrets to store your API key. Once you've configured the Weaviate secret, you can reproduce this flow without any changes. It will load the data from the Jeopardy dataset to Weaviate, and then query it with GraphQL. You can ingest data to Weaviate from a Kestra flow using one of the following options:

Key-value pairs, as shown in the batch_load_map task. This option is useful when you want to load simple key-value data.
An internal storage URI, as shown in the batch_load task. This option is recommended when you want to load data from a previous task in the same flow, e.g. after extracting it from a database or a file. The last task performing a Generative Search is currently disabled, as it requires an OpenAI API key and following the Weaviate documentation. You can enable it by removing the disabled: true line.

Download

Json To Ion

Batch Create

Query

More Related Blueprints

Ingest AI API

Generate an SEO summary of an article using HTTP REST API and OpenAI plugins

Ingest AI API

Extract data from a CSV file, load it in batch to Weaviate and query it with GraphQL

New to Kestra?

Use blueprints to kickstart your first workflows.

Get started with Kestra