GraalVM FileTransform
CertifiedTransform rows with Python on GraalVM
GraalVM FileTransform
Transform rows with Python on GraalVM
Streams rows from from (kestra:// URI, map, or list), lets Python mutate row, and writes the result as an ION file. Set concurrent to parallelize (order not preserved). Set row = None to drop a record; set rows array to emit multiple rows.
type: io.kestra.plugin.graalvm.python.FileTransformExamples
id: transformPython
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: https://dummyjson.com/carts/1
- id: jsonToIon
type: io.kestra.plugin.serdes.json.JsonToIon
from: "{{outputs.download.uri}}"
- id: transformPython
type: io.kestra.plugin.graalvm.python.FileTransform
from: "{{ outputs.jsonToIon.uri }}"
script: |
if row['id'] == 666:
# remove un-needed row
row = None
else:
# remove the 'products' column
row['products'] = None
# add a 'totalItems' column
row['totalItems'] = row['totalProducts'] * row['totalQuantity']Properties
from *Requiredstring
Source file containing rows to transform
Accepts a kestra:// internal storage URI, map, or list; rows stream into the script before being rewritten as ION
script *Requiredstring
Script body to execute
Template-rendered source code run by GraalVM in the selected language; flow variables are resolved before execution
concurrent integerstring
Parallel transformations to execute
Number of concurrent workers; ordering is not preserved when set and execution defaults to sequential when null
pluginDefaultsRef Non-dynamicstring
Reference (ref) of the pluginDefaults to apply to this task.
Outputs
uri string
uriURI of a temporary result file
The file will be serialized as an ION file.
Metrics
records counter
countNumber of records or entities processed by the Python script. This includes both modified and filtered rows from the input file.
