Query data from Druid and load it to Redshift

Move analytical data out of Apache Druid and into Amazon Redshift with a single declarative pipeline. This Druid to Redshift ETL blueprint queries a Druid datasource over JDBC, serializes the result set to CSV, stages it on Amazon S3, and loads it into a Redshift table using a native COPY command. It solves the common warehousing gap where real-time event data living in Druid needs to land in a columnar warehouse for BI, reporting, and joins with the rest of your data.

How it works

1. query_druid (io.kestra.plugin.jdbc.druid.Query) connects to Druid over the Avatica JDBC endpoint and runs a SQL SELECT against the wikipedia datasource. It uses fetchType: STORE so the full result set is written to Kestra internal storage as an Ion file rather than held in memory.
2. write_to_csv (io.kestra.plugin.serdes.csv.IonToCsv) reads that Ion output and converts it into a CSV file.
3. upload (io.kestra.plugin.aws.s3.Upload) pushes the CSV to an S3 bucket at wikipedia/input/wikipedia.csv, authenticating with AWS access keys.
4. create_table (io.kestra.plugin.jdbc.redshift.Query) runs a create table if not exists statement so the target wikipedia table is present and idempotent.
5. insert_into_redshift (io.kestra.plugin.jdbc.redshift.Query) issues a Redshift COPY command that bulk loads the staged CSV from S3 into the table.

What you get

• A repeatable Druid to Redshift data movement pipeline defined entirely in YAML.
• Efficient bulk loading via Redshift COPY instead of slow row-by-row inserts.
• Idempotent table creation that is safe to re-run.
• S3 as a durable staging layer between source and warehouse.

Who it's for

• Data engineers consolidating real-time Druid metrics into a central warehouse.
• Analytics teams that need Druid data joinable in Redshift for BI tools.
• Platform teams standardizing cross-database ETL on a single orchestrator.

Why orchestrate this with Kestra

Druid and Redshift each ship a query engine, but neither can coordinate a multi-system flow that spans both plus S3 and a CSV conversion step. Kestra glues them together: define the pipeline declaratively in YAML, add event or schedule triggers, attach retries to flaky network or warehouse calls, and capture full execution lineage and outputs for every run. You get observability and recovery across the whole Druid to S3 to Redshift path, not just inside one database.

Prerequisites

• A reachable Apache Druid instance with the wikipedia example datasource loaded (see the Druid tutorials).
• An Amazon S3 bucket for staging and an Amazon Redshift cluster.
• Kestra running locally or in your environment.

Secrets

Set these as Kestra secrets:

• AWS_ACCESS_KEY_ID: AWS access key for the S3 upload.
• AWS_SECRET_ACCESS_KEY: AWS secret key for the S3 upload.
• REDSHIFT_USER: Redshift database username.
• REDSHIFT_PASSWORD: Redshift database password.

Quick start

1. Add this blueprint to your Kestra instance.
2. Configure the four secrets above.
3. Replace the placeholder Druid url, S3 bucket, Redshift url, and the COPY credentials with your own values.
4. Execute the flow and confirm rows land in the Redshift wikipedia table.

How to extend

• Swap the SELECT in query_druid for your own Druid datasource and columns.
• Add a Schedule or event trigger to run the pipeline on a cadence.
• Insert a transformation step (for example DuckDB or Python) between the CSV write and the upload.
• Parameterize the bucket, key, and table name with inputs for reuse across datasets.

Links

• Plugin documentation: JDBC Druid
• Plugin documentation: AWS
• Plugin documentation: JDBC Redshift