Blueprints
Extract data from a REST API, process it in Python with Polars in a Docker container, then run DuckDB query and preview results as a table in the Outputs tab
About this blueprint
API DuckDB Outputs Python SQL
This flow will download a file from a REST API, process it with Python and SQL, and store the result in the internal storage.
- the
apihttp Request task uses a public API — to interact with a private API endpoint, check the task documentation for examples on how to authenticate your request - the
pythontask runs in a separate Docker container and installspolarsbefore starting the script - the DuckDB query task uses data from a previous task and outputs the query result to internal storage.
The Outputs tab provides a well-formatted table with the query results.
yaml
id: api-python-sql
namespace: blueprint
tasks:
- id: api
type: io.kestra.plugin.fs.http.Request
uri: https://dummyjson.com/products
- id: python
type: io.kestra.plugin.scripts.python.Script
docker:
image: python:slim
beforeCommands:
- pip install polars
warningOnStdErr: false
script: |
import polars as pl
data = {{outputs.api.body | jq('.products') | first}}
df = pl.from_dicts(data)
df.glimpse()
df.select(["brand", "price"]).write_csv("{{outputDir}}/products.csv")
- id: sqlQuery
type: io.kestra.plugin.jdbc.duckdb.Query
inputFiles:
in.csv: "{{ outputs.python.outputFiles['products.csv'] }}"
sql: |
SELECT brand, round(avg(price), 2) as avg_price
FROM read_csv_auto('{{workingDir}}/in.csv', header=True)
GROUP BY brand
ORDER BY avg_price DESC;
store: true