Convert an ION file into Parquet.
type: "io.kestra.plugin.serdes.parquet.iontoparquet"Read a CSV file, transform it and store the transformed data as a parquet file.
id: ion_to_parquet
namespace: company.team
tasks:
- id: download_csv
type: io.kestra.plugin.core.http.Download
description: salaries of data professionals from 2020 to 2023 (source ai-jobs.net)
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/salaries.csv
- id: avg_salary_by_job_title
type: io.kestra.plugin.jdbc.duckdb.Query
inputFiles:
data.csv: "{{ outputs.download_csv.uri }}"
sql: |
SELECT
job_title,
ROUND(AVG(salary),2) AS avg_salary
FROM read_csv_auto('{{ workingDir }}/data.csv', header=True)
GROUP BY job_title
HAVING COUNT(job_title) > 10
ORDER BY avg_salary DESC;
store: true
- id: result
type: io.kestra.plugin.serdes.parquet.IonToParquet
from: "{{ outputs.avg_salary_by_job_title.uri }}"
schema: |
{
"type": "record",
"name": "Salary",
"namespace": "com.example.salary",
"fields": [
{"name": "job_title", "type": "string"},
{"name": "avg_salary", "type": "double"}
]
}
Source file URI
Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.
GZIPUNCOMPRESSEDSNAPPYGZIPZSTDThe compression to used
yyyy-MM-dd[XXX]Format to use when parsing date
yyyy-MM-dd'T'HH:mm[:ss][.SSSSSS][XXX]Format to use when parsing datetime
Default value is yyyy-MM-dd'T'HH: mm[: ss][.SSSSSS]XXX
.Character to recognize as decimal point (e.g. use ‘,’ for European data).
Default value is '.'
1048576Max dictionary page size
["f","false","disabled","0","off","no",""]Values to consider as False
falseTry to infer all fields
If true, we try to infer all fields with trueValues, trueValues & nullValues.If false, we will infer bool & null only on field declared on schema as null and bool.
["","#N/A","#N/A N/A","#NA","-1.#IND","-1.#QNAN","-NaN","1.#IND","1.#QNAN","NA","n/a","nan","null"]Values to consider as null
100Number of row that will be scanned while inferring. The bigger it is, the more precise the output schema will be.
Only use when 'schema' property is empty
1048576Target page size
V2V1V2Target row group size
134217728Target row group size
The avro schema associated to the data
If empty the task will try to infer the schema from current data, you can use 'numberOfRowsToScan' property if needed
falseWhether to consider a field present in the data but not declared in the schema as an error
Default value is false
HH:mm[:ss][.SSSSSS][XXX]Format to use when parsing time
Etc/UTCTimezone to use when no timezone can be parsed on the source.
If null, the timezone will be UTC Default value is system timezone
["t","true","enabled","1","on","yes"]Values to consider as True
uriURI of a temporary result file