Blueprints

Ingest Zendesk data into Weaviate using dlt

Source

yaml
id: dlt-zendesk-to-weaviate
namespace: company.team

tasks:
  - id: dlt_pipeline
    type: io.kestra.plugin.scripts.python.Script
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
    containerImage: python:3.11
    beforeCommands:
      - pip install dlt[weaviate]
      - dlt --non-interactive init zendesk weaviate
    warningOnStdErr: false
    env:
      SOURCES__ZENDESK__ZENDESK_SUPPORT__CREDENTIALS__PASSWORD: "{{ secret('ZENDESK_PASSWORD') }}"
      SOURCES__ZENDESK__ZENDESK_SUPPORT__CREDENTIALS__SUBDOMAIN: "{{ secret('ZENDESK_SUBDOMAIN') }}"
      SOURCES__ZENDESK__ZENDESK_SUPPORT__CREDENTIALS__EMAIL: "{{ secret('ZENDESK_EMAIL') }}"
      DESTINATION__WEAVIATE__CREDENTIALS__URL: "{{ secret('WEAVIATE_URL') }}"
      DESTINATION__WEAVIATE__CREDENTIALS__API_KEY: "{{ secret('WEAVIATE_API_KEY') }}"
    script: |
      import dlt
      from zendesk import zendesk_support

      pipeline = dlt.pipeline(
          pipeline_name="zendesk_pipeline",
          destination="weaviate",
          dataset_name="zendesk",
      )

      zendesk_source = zendesk_support(load_all=False)
      tickets = zendesk_source.tickets

      load_info = pipeline.run(
          weaviate_adapter(
              tickets,
              vectorize=["subject", "description"],
          )
      )

About this blueprint

Ingest

This flow demonstrates how to extract data from Zendesk into Weaviate using dlt. The entire workflow logic is contained in a single Python script that uses the dlt Python library to ingest the data into Weaviate. The credentials to access the Zendesk API and Weaviate are stored using Secret.

Script

Docker

More Related Blueprints

New to Kestra?

Use blueprints to kickstart your first workflows.

Get started with Kestra