Python - GraalVM FileTransform
CertifiedTransform rows with Python on GraalVM
Python - GraalVM FileTransform
Transform rows with Python on GraalVM
Streams rows from from (kestra:// URI, map, or list), lets Python mutate row, and writes the result as an ION file. Set concurrent to parallelize (order not preserved). Set row = None to drop a record; set rows to a list to emit several rows instead of row (an empty list drops the record). File access through Python is limited to the task working directory and the read-only directories from the allowed-paths plugin configuration of this task type, except for native modules (see below). Supports C-extension-backed stdlib modules such as ssl, sqlite3, and lzma, which requires enabling native access at the GraalVM engine level. Standard OS process APIs (os.system, subprocess) stay blocked, but this grant cannot be scoped down further: a script that reaches native code directly (e.g. ctypes) can still execute arbitrary OS commands, read or write any file the worker can access, or manipulate process memory. Native modules also open files themselves, outside the working directory restriction: sqlite3.connect() can read or create a database file anywhere the worker can. Unlike Kestra's Script/Shell tasks, which typically run in an isolated container or process via a TaskRunner, this code runs inline in the worker JVM process itself, so it has direct access to the worker process's own memory, file descriptors, and any secrets or other task state resident in that JVM. Only run scripts from users already trusted with the flow's credentials and infrastructure, as with any script task.
type: io.kestra.plugin.graalvm.python.FileTransformExamples
id: transformPython
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: https://dummyjson.com/carts/1
- id: jsonToIon
type: io.kestra.plugin.serdes.json.JsonToIon
from: "{{outputs.download.uri}}"
- id: transformPython
type: io.kestra.plugin.graalvm.python.FileTransform
from: "{{ outputs.jsonToIon.uri }}"
script: |
if row['id'] == 666:
# remove un-needed row
row = None
else:
# remove the 'products' column
row['products'] = None
# add a 'totalItems' column
row['totalItems'] = row['totalProducts'] * row['totalQuantity']Transform rows and control output column order for downstream tasks
id: transformPythonColumns
namespace: company.team
tasks:
- id: download
type: io.kestra.plugin.core.http.Download
uri: https://dummyjson.com/carts/1
- id: jsonToIon
type: io.kestra.plugin.serdes.json.JsonToIon
from: "{{outputs.download.uri}}"
- id: transform
type: io.kestra.plugin.graalvm.python.FileTransform
from: "{{ outputs.jsonToIon.uri }}"
columns:
- id
- totalItems
script: |
row['totalItems'] = row['totalProducts'] * row['totalQuantity']
Properties
from *string
Source file containing rows to transform
Accepts a kestra:// internal storage URI, map, or list; rows stream into the script before being rewritten as ION
script *string
Script body to execute
Template-rendered source code run by GraalVM in the selected language; flow variables are resolved before execution
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150columns array
Columns to include in the output and their order
When set, each output row contains exactly these columns in the specified order. A column that is not present in a given row is written as null so that every row has the same shape. Columns present in the row but not listed here are dropped. When not set or empty, rows are written as is. This is useful when downstream tasks (e.g. JDBC Batch) depend on a fixed column count and order.
concurrent integerstring
Parallel transformations to execute
Number of concurrent workers; ordering is not preserved when set and execution defaults to sequential when null
options object
Advanced GraalVM context options
Maps directly to GraalVM's Context.Builder#options: one entry per option key/value, for example python.WarnOptions or js.ecmascript-version.
Keys that would weaken the sandbox already enforced by this task are rejected at execution time: python.PosixModuleBackend, any key related to host access or class loading, and any key prefixed with engine. or sandbox..
Outputs
uri string
uriURI of a temporary result file
The file will be serialized as an ION file.
Metrics
records counter
countNumber of records or entities processed by the Python script. This includes both modified and filtered rows from the input file.
