Ingest Zendesk data into DuckDB using dlt

Pull Zendesk Support data into a local DuckDB warehouse using the open source dlt (data load tool) Python library, fully orchestrated by Kestra. This blueprint solves the classic support analytics problem: ticket data lives behind the Zendesk API and is hard to query or join with other datasets. The flow turns that API into an analytics-ready DuckDB dataset in a single, reproducible Python task, so you can run SQL over your tickets without standing up heavy infrastructure.

How it works

The flow contains one task, dlt_pipeline, of type io.kestra.plugin.scripts.python.Script. It runs on a io.kestra.plugin.scripts.runner.docker.Docker task runner using the python:3.11 container image, so the environment is isolated and disposable on every run.

1. beforeCommands install the dlt DuckDB extra with pip install dlt[duckdb] and scaffold the verified Zendesk source with dlt --non-interactive init zendesk duckdb.
2. Zendesk credentials are passed as environment variables that dlt reads automatically (SOURCES__ZENDESK__ZENDESK_SUPPORT__CREDENTIALS__*), each sourced from a Kestra {{ secret(...) }} so nothing is hardcoded.
3. The inline Python script builds a dlt.pipeline named zendesk_pipeline with a duckdb destination and a zendesk dataset, calls zendesk_support(load_all=False), selects the tickets resource, and runs pipeline.run(tickets) to load the data.

What you get

• A DuckDB dataset named zendesk containing your support tickets.
• Schema inference, normalization, and load handled by dlt, not by hand-written glue.
• A clean, containerized run with no local Python setup to maintain.
• Centralized, encrypted credential handling through Kestra Secrets.

Who it's for

• Support and customer experience analysts who want to query tickets with SQL.
• Data engineers prototyping an ELT pipeline before promoting it to production.
• Analytics engineers building local models on top of Zendesk data with DuckDB.

Why orchestrate this with Kestra

dlt loads data, but it does not schedule, retry, or observe itself. Kestra wraps the pipeline in declarative YAML, runs it in a reproducible Docker container, and lets you attach event triggers (cron, webhook, or upstream flow completion) so loads happen on a schedule or in reaction to events. You get automatic retries on transient API failures, full execution logs and lineage in the UI, and centralized secret management, the operational layer the dlt CLI alone cannot provide.

Prerequisites

• A running Kestra instance with the Docker task runner available.
• A Zendesk account with API access enabled.

Secrets

Configure these Kestra Secrets before running:

• ZENDESK_SUBDOMAIN: your Zendesk subdomain.
• ZENDESK_EMAIL: the email of the API user.
• ZENDESK_PASSWORD: the password or API token for that user.

Quick start

1. Add the three secrets above to your Kestra instance.
2. Copy this blueprint into a flow and adjust the namespace if needed.
3. Execute the flow and watch the dlt_pipeline task logs.
4. Inspect the resulting zendesk dataset in your DuckDB file.

How to extend

• Set load_all=True or select more resources (users, organizations) instead of only tickets.
• Swap the duckdb destination for another dlt destination such as BigQuery, Snowflake, or Postgres.
• Add a io.kestra.plugin.core.trigger.Schedule trigger for daily incremental loads.
• Chain a downstream dbt or SQL task to model the loaded tables.

Links

• Plugin docs: plugin-scripts
• Orchestrate DuckDB with Kestra