Serialization CsvToIon
CertifiedConvert a CSV file to the Amazon ION format
Serialization CsvToIon
Convert a CSV file to the Amazon ION format
Supports configurable field separator, text delimiter, charset, and header detection. The value \N is treated as null in any field. Use onBadLines to control error handling for malformed rows. A leading UTF-8 byte-order mark is stripped automatically. Trailing unnamed header columns (e.g. from a trailing field separator) are dropped by default; set onEmptyHeader to RENAME to keep them with generated names (col_0, col_1, ...) instead.
type: io.kestra.plugin.serdes.csv.CsvToIonExamples
Convert a CSV file to the Amazon ION format.
id: csv_to_ion
namespace: company.team
tasks:
- id: http_download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/products.csv
- id: to_ion
type: io.kestra.plugin.serdes.csv.CsvToIon
from: "{{ outputs.http_download.uri }}"
Properties
from *Requiredstring
Source file URI
Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.
allowExtraCharsAfterClosingQuote booleanstring
falseAllow extra characters after a closing quote
charset string
UTF-8The name of a supported charset
fieldSeparator string
,The field separator character
header booleanstring
trueSpecifies if the first line should be the header
maxBufferSize integerstring
16777216Maximum CSV parser buffer size (bytes)
maxFieldSize integerstring
16777216Maximum field size (characters)
onBadLines string
ERRORERRORWARNSKIPHow to handle bad lines (e.g., a line with too many fields)
onEmptyHeader string
DROPDROPRENAMEHow to handle columns whose header name is empty
DROP (default) removes trailing unnamed columns. RENAME keeps every column and names unnamed ones col_0, col_1, ... to avoid losing data.
skipEmptyRows booleanstring
falseSpecifies if empty rows should be skipped
skipRows integerstring
0Number of rows to skip at the start of the file, before the header row is read
Rows are counted at the CSV-record level, not by raw physical line: a quoted field spanning several physical lines counts as a single row. When header is true, skipped rows are removed first and the header is then read from the next row; when header is false, the same number of rows is removed from the start of the data. A negative value is treated as 0. Rows dropped by onBadLines (WARN or SKIP) never reach the skip counter, so a malformed row inside the skip window does not count against skipRows.
textDelimiter string
"The text delimiter character
Outputs
size integer
0The number of records converted
uri string
uriURI of a temporary result file
Metrics
records counter
Number of records converted
