Serialization IonToParquet
CertifiedConvert an Ion file to the Parquet format
Serialization IonToParquet
Convert an Ion file to the Parquet format
An Avro schema is required to define column types; if none is provided, one is inferred by scanning up to numberOfRowsToScan rows. Supports configurable compression (default: GZIP), Parquet format version, row group size, page size, and dictionary page size.
type: io.kestra.plugin.serdes.parquet.IonToParquetExamples
Read a CSV file, transform it and store the transformed data as a parquet file.
id: ion_to_parquet
namespace: company.team
tasks:
- id: download_csv
type: io.kestra.plugin.core.http.Download
description: salaries of data professionals from 2020 to 2023 (source ai-jobs.net)
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/salaries.csv
- id: avg_salary_by_job_title
type: io.kestra.plugin.jdbc.duckdb.Query
inputFiles:
data.csv: "{{ outputs.download_csv.uri }}"
sql: |
SELECT
job_title,
ROUND(AVG(salary),2) AS avg_salary
FROM read_csv_auto('{{ workingDir }}/data.csv', header=True)
GROUP BY job_title
HAVING COUNT(job_title) > 10
ORDER BY avg_salary DESC;
store: true
- id: result
type: io.kestra.plugin.serdes.parquet.IonToParquet
from: "{{ outputs.avg_salary_by_job_title.uri }}"
schema: |
{
"type": "record",
"name": "Salary",
"namespace": "com.example.salary",
"fields": [
{"name": "job_title", "type": "string"},
{"name": "avg_salary", "type": "double"}
]
}
Properties
from *Requiredstring
Source file URI
Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.
compressionCodec string
GZIPUNCOMPRESSEDSNAPPYGZIPZSTDCompression codec
dateFormat string
yyyy-MM-dd[XXX]Format to use when parsing date
datetimeFormat string
yyyy-MM-dd'T'HH:mm[:ss][.SSSSSS][XXX]Format to use when parsing datetime
Default value is yyyy-MM-dd'T'HH: mm[: ss][.SSSSSS][XXX]
decimalSeparator string
.Character to recognize as decimal point (e.g. use ‘,’ for European data)
Default value is '.'
dictionaryPageSize integerstring
1048576Max dictionary page size
falseValues array
["f","false","disabled","0","off","no",""]Values to consider as False
inferAllFields booleanstring
falseTry to infer all fields
If true, schema inference scans all rows (ignoring numberOfRowsToScan) and attempts to infer all field types using trueValues, falseValues, and nullValues. This prevents fields that are null in the first scanned rows from being typed as NULL. If false, only the first numberOfRowsToScan rows are scanned, and booleans/nulls are inferred only on fields declared in the schema as null or bool.
nullValues array
["","#N/A","#N/A N/A","#NA","-1.#IND","-1.#QNAN","-NaN","1.#IND","1.#QNAN","NA","n/a","nan","null"]Values to consider as null
numberOfRowsToScan integerstring
100Number of rows that will be scanned while inferring. The more rows scanned, the more precise the output schema will be
Only use when the 'schema' property is empty. Ignored for schema inference when inferAllFields is true — in that case, all rows are scanned.
onBadLines string
ERRORERRORWARNSKIPHow to handle bad records (e.g., null values in non-nullable fields or type mismatches)
Can be ERROR, WARN, or SKIP.
pageSize integerstring
1048576Target page size
parquetVersion string
V2V1V2Parquet format version
pluginDefaultsRef Non-dynamicstring
Reference (ref) of the pluginDefaults to apply to this task.
rowGroupSize integerstring
134217728Target row group size
schema string
The avro schema associated with the data
If empty, the task will try to infer the schema from the current data; use the 'numberOfRowsToScan' property if needed
strictSchema booleanstring
falseWhether to consider a field present in the data but not declared in the schema as an error
Default value is false
timeFormat string
HH:mm[:ss][.SSSSSS][XXX]Format to use when parsing time
timeZoneId string
Etc/UTCTimezone to use when no timezone can be parsed on the source
If null, the timezone defaults to UTC. Default value is the system timezone
trueValues array
["t","true","enabled","1","on","yes"]Values to consider as True
Outputs
size integer
0The number of records converted
uri string
uriURI of a temporary result file
Metrics
records counter
Number of records converted
