Analyze salaries of data professionals using a SQL query in DuckDB

Run fast, in-process SQL analytics on a remote CSV without standing up a database server. This blueprint downloads a public salaries dataset over HTTP, queries it with DuckDB to compute the average salary per job title, and exports the aggregated result back to a clean CSV file. It is a practical pattern for ad-hoc analytics, data profiling, and lightweight ETL where you want columnar SQL performance directly over flat files.

How it works

1. download_csv uses io.kestra.plugin.core.http.Download to fetch a public salaries.csv dataset (data professional salaries, 2020 to 2023, source ai-jobs.net) and stores it in Kestra internal storage.
2. avg_salary_by_job_title uses io.kestra.plugin.jdbc.duckdb.Query to mount the downloaded file via inputFiles as data.csv, then runs SQL with read_csv_auto('{{ workingDir }}/data.csv', header=True). It groups by job_title, keeps titles with more than 10 records via HAVING COUNT(job_title) > 10, and orders by descending average salary. Results are persisted with fetchType: STORE.
3. result uses io.kestra.plugin.serdes.csv.IonToCsv to convert the stored Ion output into a downloadable CSV artifact.

What you get

• Average salary per job title, filtered to roles with meaningful sample size.
• A reusable, serverless SQL-over-CSV pattern with no database to provision.
• A final CSV artifact ready to share, download, or pass downstream.

Who it's for

• Data analysts and engineers who want quick SQL over files.
• Analytics teams prototyping transformations before committing to a warehouse.
• Anyone profiling or aggregating CSV, Parquet, or JSON data sources.

Why orchestrate this with Kestra

DuckDB is an embedded engine: it has no scheduler, no retry logic, and no built-in way to react to events. Kestra wraps the query in a declarative YAML workflow so you can attach event or schedule triggers, add retries on transient HTTP failures, capture full execution lineage and outputs, and chain the download, query, and export steps with explicit dependencies. You get production orchestration around an engine that, on its own, only runs a single query.

Prerequisites

• A running Kestra instance with the DuckDB JDBC plugin available.
• Outbound network access to the dataset URL.

Secrets

This flow uses no secrets. The dataset is fetched from a public URL and DuckDB runs in-process, so no credentials are required. If you point it at a private source, add credentials with {{ secret('NAME') }}.

Quick start

1. Add this flow to a Kestra namespace.
2. Execute it from the UI.
3. Open the result task output and download the generated CSV.

How to extend

• Swap the uri to your own CSV, Parquet, or JSON file and adjust the SQL.
• Add a Schedule or event trigger to refresh results automatically.
• Replace IonToCsv with a JSON or Parquet serializer, or load results into a warehouse.
• Add retries and notifications on failure for production runs.

Links

• DuckDB plugin docs
• DuckDB orchestration with Kestra