Fetch data from Apache Druid and transform it in Python with Pandas

Move analytical data out of Apache Druid and into a Python workflow without glue scripts. This blueprint runs a SQL query against Druid over its Avatica JDBC endpoint, serializes the result set to CSV, and hands the file to a Pandas script for downstream analysis. It solves the common gap between a real-time analytics database and the Python tooling that data scientists and analysts actually use for transformation, feature engineering, and reporting.

How it works

1. The query_druid task (io.kestra.plugin.jdbc.druid.Query) connects to Druid through the Avatica JDBC URL and runs a SQL SELECT over the wikipedia datasource. With fetchType: STORE, the full result set is written to Kestra internal storage as an Ion file and exposed as a URI.
2. The write_to_csv task (io.kestra.plugin.serdes.csv.IonToCsv) reads that Ion output and converts it to a CSV file, producing a clean tabular artifact.
3. The process_using_pandas task (io.kestra.plugin.scripts.python.Script) declares pandas as a dependency, reads the CSV with pd.read_csv, and runs your analysis logic.

What you get

• A reproducible Druid to Pandas data path defined entirely in YAML.
• Result sets streamed to internal storage instead of held in memory.
• A CSV artifact that any Python, R, or BI tool can consume.
• A Python step with isolated, declarative dependency management.

Who it's for

• Data analysts pulling Druid metrics into notebooks and reports.
• Data engineers building scheduled extracts from real-time analytics.
• Data scientists who need Druid data in a Pandas DataFrame.

Why orchestrate this with Kestra

Druid is excellent at serving sub-second analytical queries, but it has no native scheduler to extract data, convert formats, and trigger downstream Python on a cadence or in response to an event. Kestra fills that gap: event and schedule triggers kick off the pipeline, per-task retries handle transient JDBC failures, execution outputs give you lineage from query to CSV to script, and the whole flow is declarative YAML you can version and review.

Prerequisites

• A reachable Apache Druid instance exposing the Avatica SQL JDBC endpoint.
• A Kestra instance with the JDBC Druid, SerDes, and Python script plugins.
• Docker available for the Python script task runner.

Secrets

This blueprint connects to a local Druid endpoint and references no secret() values. When pointing at a secured Druid deployment, move credentials into Kestra secrets and inject them into the JDBC url, username, and password properties rather than hardcoding them.

Quick start

1. Add the flow to your Kestra instance.
2. Set the url property to your Druid Avatica endpoint.
3. Adjust the SQL to target your own datasource and columns.
4. Execute the flow and inspect the Pandas script output.

How to extend

• Replace df.head() with real transformations, aggregations, or feature engineering.
• Write the processed DataFrame to a database, object store, or data warehouse.
• Add a schedule or event trigger to run the extract automatically.
• Parameterize the SQL and datasource with flow inputs.

Links

• Plugin documentation: JDBC Druid
• Apache Druid tutorials