Ingest Zendesk data into Weaviate using dlt

Ingest Zendesk support tickets into a Weaviate vector database using the dlt (data load tool) Python library, orchestrated by Kestra. This blueprint extracts tickets from the Zendesk Support API and loads them into Weaviate as vectorized objects, so the ticket subject and description become searchable by meaning. It is a ready-made starting point for building semantic search, retrieval-augmented generation (RAG), and support analytics on top of your Zendesk data without stitching together a custom EL pipeline.

How it works

The flow runs a single task, dlt_pipeline, of type io.kestra.plugin.scripts.python.Script. The task executes on a io.kestra.plugin.scripts.runner.docker.Docker task runner using the python:3.11 container image, so the runtime is fully isolated and reproducible.

• beforeCommands install the connector with pip install dlt[weaviate] and scaffold the source and destination with dlt --non-interactive init zendesk weaviate.
• Zendesk and Weaviate credentials are injected as environment variables using dlt's SOURCES__ and DESTINATION__ naming convention, each resolved from {{ secret(...) }}.
• The inline Python script builds a dlt pipeline named zendesk_pipeline with destination="weaviate" and dataset_name="zendesk", pulls the tickets resource from zendesk_support(load_all=False), and runs it through the weaviate_adapter with vectorize=["subject", "description"].

What you get

• Zendesk tickets loaded into Weaviate, ready for vector search.
• Subject and description fields vectorized for semantic retrieval and RAG.
• A schema managed automatically by dlt, no manual DDL.
• A containerized, repeatable run that captures logs and execution history in Kestra.

Who it's for

• Data engineers building EL pipelines from SaaS APIs into a vector store.
• AI and ML engineers assembling a knowledge base for RAG over support content.
• Support and operations teams that want semantic search across historical tickets.

Why orchestrate this with Kestra

dlt knows how to move and vectorize the data, but it does not schedule, observe, or recover the pipeline on its own. Kestra adds event and schedule triggers so loads can fire on a cron or in response to an upstream event, automatic retries on transient API failures, full execution history and logs for lineage and debugging, and centralized secret management. The whole pipeline stays declarative YAML in version control, and the Docker runner guarantees the exact Python environment on every run.

Prerequisites

• A Kestra instance with Docker available to the task runner.
• A Zendesk account with API access and a reachable Weaviate instance.
• The secrets below configured in Kestra.

Secrets

• ZENDESK_PASSWORD
• ZENDESK_SUBDOMAIN
• ZENDESK_EMAIL
• WEAVIATE_URL
• WEAVIATE_API_KEY

Quick start

1. Add this flow to a Kestra namespace.
2. Configure the five secrets listed above.
3. Execute the flow and watch the dlt_pipeline task logs as dlt installs, initializes, extracts, and loads.
4. Query Weaviate to confirm the zendesk dataset is populated with vectorized tickets.

How to extend

• Set load_all=True to ingest more Zendesk resources beyond tickets (users, organizations, satisfaction ratings).
• Adjust the vectorize list to embed additional fields.
• Swap the destination to BigQuery, Postgres, Snowflake, or DuckDB by changing the dlt destination and credentials.
• Add a Schedule trigger for periodic loads or a Flow trigger to chain a downstream RAG indexing flow.

Links

• Plugin Scripts documentation