Serialization ProtobufToIon
CertifiedConvert a Protobuf file into Amazon ION
Serialization ProtobufToIon
Convert a Protobuf file into Amazon ION
The plugin reads one or more Protobuf messages from a binary file or stream, decodes them using a provided descriptor and message type name, and then serializes the result as ION data based on ProtoJSON Format. It requires the following information regarding the Protobuf message:
- A descriptor file (
.desc), generated with--descriptor_set_out, that contains the compiled Protobuf message definitions. - A type name (e.g.,
com.company.Product) corresponding to the root message to decode.
Here's an example of how to generate a descriptor file from a Protobuf file using protoc:
protoc --proto_path=src/main/proto --descriptor_set_out=products.desc src/main/proto/products.proto
type: io.kestra.plugin.serdes.protobuf.ProtobufToIonExamples
Convert a Protobuf file to the Amazon ION format.
id: protobuf_to_ion
namespace: company.team
tasks:
- id: http_download
type: io.kestra.plugin.core.http.Download
uri: https://example.com/data/products.pb
- id: to_ion
type: io.kestra.plugin.serdes.protobuf.ProtobufToIon
from: "{{ outputs.http_download.uri }}"
descriptorFile: "kestra:///path/to/proto.desc"
typeName: com.company.Product
Properties
descriptorFile *string
Protobuf Descriptor File
Protobuf descriptor file containing message definitions.
Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.
from *string
Source file URI
The URI of the input Protobuf file to read.
Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.
typeName *string
Fully qualified Protobuf message type name
For example: com.company.Product.
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150delimited booleanstring
falseIs the input a stream of length-delimited messages?
If true, the input file is expected to contain multiple length-delimited Protobuf messages. If false, it will be parsed as a single Protobuf message.
errorOnUnknownFields booleanstring
falseWhether to error on unknown fields
If true, an error will be thrown if the input contains unknown fields.
Outputs
size integer
0The number of records converted
uri string
uriURI of a temporary result file
Metrics
records counter
Number of Protobuf messages converted
