FileTransform FileTransform

type: "io.kestra.plugin.scripts.jython.FileTransform"

Transform ion format file from kestra with a groovy script.

This allow you to transform the data previously loaded by kestra as you need.

Take a ion format file from kestra and iterate row per row.
Each row will populate a row global variable, you need to alter this variable that will be saved on output file.
if you set the row to null, the row will be skipped
You can create a variables rows to return many rows for a single row.

# Examples

Transform with file from internal storage

id: "file_transform"
type: "io.kestra.plugin.scripts.jython.FileTransform"
from: "{{ outputs['avro-to-gcs'] }}"
script: |
  logger.info('row: {}', row)

  if row['name'] == 'richard': 
    row = None
  else: 
    row['email'] = row['name'] + '@kestra.io'

Transform with file from json string

id: "file_transform"
type: "io.kestra.plugin.scripts.jython.FileTransform"
from: "[{"name":"jane"}, {"name":"richard"}]"
script: |
  logger.info('row: {}', row)

  if row['name'] == 'richard': 
    row = None
  else: 
    row['email'] = row['name'] + '@kestra.io'

# Properties

# concurrent

  • Type: integer
  • Dynamic:
  • Required:
  • Minimum: >= 2

Number of concurrent parallels transform

Take care that the order is not respected if you use parallelism

# from

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

Source file of row to transform

Can be an internal storage uri, a map or a list.

# script

  • Type: string
  • Dynamic: ✔️
  • Required:

A full script

# Outputs

# uri

  • Type: string

URI of a temporary result file

The file will be serialized as ion file.