Serialization CsvToIon
CertifiedConvert a CSV file to the Amazon ION format
Serialization CsvToIon
Convert a CSV file to the Amazon ION format
Supports configurable field separator, text delimiter, charset, and header detection. The value \N is treated as null in any field. Use onBadLines to control error handling for malformed rows. A leading UTF-8 byte-order mark is stripped automatically. Trailing unnamed header columns (e.g. from a trailing field separator) are dropped by default; set onEmptyHeader to RENAME to keep them with generated names (col_0, col_1, ...) instead.
type: io.kestra.plugin.serdes.csv.CsvToIonExamples
Convert a CSV file to the Amazon ION format.
id: csv_to_ion
namespace: company.team
tasks:
- id: http_download
type: io.kestra.plugin.core.http.Download
uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/products.csv
- id: to_ion
type: io.kestra.plugin.serdes.csv.CsvToIon
from: "{{ outputs.http_download.uri }}"
Properties
from *string
Source file URI
Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.
allowExtraCharsAfterClosingQuote booleanstring
falseAllow extra characters after a closing quote
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150charset string
UTF-8The name of a supported charset
fieldSeparator string
,The field separator character
header booleanstring
trueSpecifies if the first line should be the header
maxBufferSize integerstring
16777216Maximum CSV parser buffer size (characters)
The FastCSV parser's maximum internal buffer size. The value must be positive and cannot exceed 2,147,483,647 characters.
maxFieldSize integerstring
16777216Maximum field size (characters)
onBadLines string
ERRORERRORWARNSKIPHow to handle bad lines (e.g., a line with too many fields)
onEmptyHeader string
DROPDROPRENAMEHow to handle columns whose header name is empty
DROP (default) removes trailing unnamed columns. RENAME keeps every column and names unnamed ones col_0, col_1, ... to avoid losing data.
skipEmptyRows booleanstring
falseSpecifies if empty rows should be skipped
skipRows integerstring
0Number of rows to skip at the start of the file, before the header row is read
Rows are counted at the CSV-record level, not by raw physical line: a quoted field spanning several physical lines counts as a single row. When header is true, skipped rows are removed first and the header is then read from the next row; when header is false, the same number of rows is removed from the start of the data. A negative value is treated as 0. Rows dropped by onBadLines (WARN or SKIP) never reach the skip counter, so a malformed row inside the skip window does not count against skipRows.
textDelimiter string
"The text delimiter character
Outputs
size integer
0The number of records converted
uri string
uriURI of a temporary result file
Metrics
records counter
Number of records converted
