Python Script

Python Script

Certified

Run inline Python script

Executes a multi-line Python script inside the default 'python: 3.13-slim' image unless overridden. Handles dependency install via UV (default) or pip with optional cache; script is saved to a temp .py file and run with the resolved interpreter. Use Commands task to run existing files.

yaml
type: io.kestra.plugin.scripts.python.Script

Execute a Python script and generate an output.

yaml
id: python_demo
namespace: company.team

tasks:
  - id: python
    type: io.kestra.plugin.scripts.python.Script
    dependencies:
      - requests
      - kestra
    script: |
      from kestra import Kestra
      import requests

      response = requests.get('https://kestra.io')
      print(response.status_code)

      Kestra.outputs({'status': response.status_code, 'text': response.text})

Install pip packages before starting a Python Script task

yaml
id: pip_packages_docker
namespace: company.team

tasks:
  - id: run_python
    type: io.kestra.plugin.scripts.python.Script
    dependencies:
      - requests
    script: |
      import requests
      import json

      response = requests.get("https://api.github.com")
      data = response.json()
      print(data)

Log messages at different log levels using Kestra logger.

yaml
id: python_logs
namespace: company.team

tasks:
  - id: python_logger
    type: io.kestra.plugin.scripts.python.Script
    allowFailure: true
    dependencies:
      - kestra
    script: |
      import time
      from kestra import Kestra

      logger = Kestra.logger()

      logger.debug("DEBUG is used for diagnostic info.")
      time.sleep(0.5)

      logger.info("INFO confirms normal operation.")
      time.sleep(0.5)

      logger.warning("WARNING signals something unexpected.")
      time.sleep(0.5)

      logger.error("ERROR indicates a serious issue.")
      time.sleep(0.5)

      logger.critical("CRITICAL means a severe failure.")

Execute a Python script with a file stored in Kestra's local storage created by a previous task.

yaml
id: pass_data_between_tasks
namespace: company.team

tasks:
  - id: download
    type: io.kestra.plugin.core.http.Download
    uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv

  - id: python
    type: io.kestra.plugin.scripts.python.Script
    script: |
      with open('{{ outputs.download.uri }}', 'r') as f:
        print(f.read())

Execute a Python script that outputs a file.

yaml
id: python_output_file
namespace: company.team

tasks:
  - id: python
    type: io.kestra.plugin.scripts.python.Script
    outputFiles:
      - "myfile.txt"
    script: |
      f = open("myfile.txt", "a")
      f.write("Hello from a Kestra task!")
      f.close()

If you want to generate files in your script to make them available for download and use in downstream tasks, you can leverage the outputFiles property as shown in the example above. Files will be persisted in Kestra's internal storage. The first task in this example creates a file 'clean_dataset.csv' and the next task can access it by leveraging the syntax {{outputs.yourTaskId.outputFiles['yourFileName.fileExtension']}}.

yaml
id: python_outputs
namespace: company.team

tasks:
  - id: clean_dataset
    type: io.kestra.plugin.scripts.python.Script
    containerImage: ghcr.io/kestra-io/pydata:latest
    outputFiles:
      - "clean_dataset.csv"
    dependencies:
      - pandas
    script: |
      import pandas as pd
      df = pd.read_csv("https://huggingface.co/datasets/kestra/datasets/raw/main/csv/messy_dataset.csv")

      # Replace non-numeric age values with NaN
      df["Age"] = pd.to_numeric(df["Age"], errors="coerce")

      # mean imputation: fill NaN values with the mean age
      mean_age = int(df["Age"].mean())
      print(f"Filling NULL values with mean: {mean_age}")
      df["Age"] = df["Age"].fillna(mean_age)
      df.to_csv("clean_dataset.csv", index=False)

  - id: read_file_from_python
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    commands:
      - head -n 10 {{ outputs.clean_dataset.outputFiles['clean_dataset.csv'] }}

Create a Python inline script that takes input using an expression

yaml
id: python_use_input_in_inline
namespace: company.team

inputs:
  - id: pokemon
    type: STRING
    defaults: pikachu

  - id: your_age
    type: INT
    defaults: 25

tasks:
  - id: inline_script
    type: io.kestra.plugin.scripts.python.Script
    description: Fetch the pokemon detail and compare its experience
    containerImage: ghcr.io/kestra-io/pydata:latest
    dependencies:
      - requests
    script: |
      import requests
      import json

      url = "https://pokeapi.co/api/v2/pokemon/{{ inputs.pokemon }}"
      response = requests.get(url)

      if response.status_code == 200:
          pokemon = json.loads(response.text)
          print(f"Base experience of {{ inputs.pokemon }} is { pokemon.get('base_experience') }")
          if pokemon.get('base_experience') > int("{{ inputs.your_age }}"):
              print("{{ inputs.pokemon }} has more base experience than your age")
          else:
              print("{{ inputs.pokemon}} is too young!")
      else:
          print(f"Failed to retrieve the webpage. Status code: {response.status_code}")

Pass an input file to a Python script

yaml
id: python_input_file
namespace: company.team

tasks:
  - id: download_file
    type: io.kestra.plugin.core.http.Download
    uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv

  - id: get_total_rows
    type: io.kestra.plugin.scripts.python.Script
    dependencies:
      - pandas
    inputFiles:
      input.csv: "{{ outputs.download_file.uri }}"
    script: |
      import pandas as pd

      # Path to your CSV file
      csv_file_path = "input.csv"

      # Read the CSV file using pandas
      df = pd.read_csv(csv_file_path)

      # Get the number of rows
      num_rows = len(df)

      print(f"Number of rows: {num_rows}")

Run a simple Python script to generate outputs and log them

yaml
id: python_generate_outputs
namespace: company.team

tasks:
  - id: generate_output
    type: io.kestra.plugin.scripts.python.Script
    packageManager: PIP
    dependencies:
      - kestra
    script: |
      from kestra import Kestra

      marks = [79, 91, 85, 64, 82]
      Kestra.outputs({"total_marks": sum(marks),"average_marks": sum(marks)/len(marks)})

  - id: log_result
    type: io.kestra.plugin.core.log.Log
    message:
      - "Total Marks: {{ outputs.generate_output.vars.total_marks }}"
      - "Average Marks: {{ outputs.generate_output.vars.average_marks }}"
Properties

Inline Python script

Python source as a multi-line string; written to a temporary .py file and executed with the resolved interpreter. For existing files, use the Commands task.

SubTypestring

A list of commands that will run before the commands, allowing to set up the environment e.g. pip install -r requirements.txt.

Defaultpython:3.13-slim

The container image to use for the script

Defaults to python: 3.13-slim. The Python version is auto-detected from the image tag when it matches the pattern python: <version> (e.g. python: 3.12, python: 3.13-slim). Tags like latest or custom images will not be detected. If version inference fails, Kestra uses Python 3.13 for dependency resolution and cache key computation, while the interpreter available in the container may differ. Set pythonVersion explicitly or use a versioned Python image tag to avoid mismatches.

SubTypestring

Python package dependencies

List of pip-compatible package specifiers (e.g. pandas==2.0.0, requests>=2.28) installed via the configured package manager before script execution.

Defaulttrue

Enable Python dependency caching

When enabled, Python dependencies will be cached across task executions. This locks dependency versions and speeds up subsequent runs by avoiding redundant installations.

SubTypestring

Additional environment variables for the current process.

Defaulttrue

Fail the task on the first command with a non-zero status.

If set to false all commands will be executed one after the other. The final state of task execution is determined by the last command. Note that this property maybe be ignored if a non compatible interpreter is specified. You can also disable it if your interpreter does not support the set -eoption.

The files to create on the working. It can be a map or a JSON object.

Each file can be defined:

  • Inline with its content
  • As a URI, supported schemes are kestra for internal storage files, file for host local files, and nsfile for namespace files.
SubTypestring
Default["/bin/sh","-c"]

Which interpreter to use.

Inject namespace files.

Inject namespace files to this task. When enabled, it will, by default, load all namespace files into the working directory. However, you can use the include or exclude properties to limit which namespace files will be injected.

Definitions
enabledbooleanstring
Defaulttrue

Whether to enable namespace files to be loaded into the working directory. If explicitly set to true in a task, it will load all Namespace Files into the task's working directory. Note that this property is by default set to true so that you can specify only the include and exclude properties to filter the files to load without having to explicitly set enabled to true.

excludearray
SubTypestring

A list of filters to exclude matching glob patterns. This allows you to exclude a subset of the Namespace Files from being downloaded at runtime. You can combine this property together with include to only inject a subset of files that you need into the task's working directory.

folderPerNamespacebooleanstring
Defaultfalse

Whether to mount file into the root of the working directory, or create a folder per namespace

ifExistsstring
DefaultOVERWRITE
Possible Values
OVERWRITEFAILWARNIGNORE

Comportment of the task if a file already exist in the working directory.

includearray
SubTypestring

A list of filters to include only matching glob patterns. This allows you to only load a subset of the Namespace Files into the working directory.

namespacesarray
SubTypestring
Default["{{flow.namespace}}"]

A list of namespaces in which searching files. The files are loaded in the namespace order, and only the latest version of a file is kept. Meaning if a file is present in the first and second namespace, only the file present on the second namespace will be loaded.

SubTypestring

List of output file names to capture after execution.

Declares files produced by the script that should be uploaded to Kestra's internal storage. Use {{ outputFiles["filename"] }} in your script body to get the resolved absolute path — this works for local runners (Process, Docker) and remote runners (Kubernetes, cloud batch). Glob patterns (e.g. *.csv) are supported for post-run collection but cannot be referenced via Pebble expressions inside the script.

DefaultUV
Possible Values
PIPUV

Package manager for Python dependencies

Package manager to use for installing Python dependencies. Options: 'UV' (default), 'PIP'. UV automatically falls back to PIP if not available.

Reference (ref) of the pluginDefaults to apply to this task.

The version of Python to use for the script

If no version is explicitly specified, the task will attempt to extract the version from the configured container image or from the local Python installation depending on the configured task runner. The version is parsed from containerImage only when it matches the pattern python: <numeric-version> (e.g. python: 3.12, python: 3.13-slim). Tags like latest or custom images (e.g. ghcr.io/kestra-io/pydata: latest) will not be detected. If it cannot determine the version, the task will default to Python 3.13 for dependency resolution and cache key computation, while the interpreter available in the container may differ. Set this property explicitly or use a versioned Python image tag to avoid version mismatches.

DefaultAUTO
Possible Values
LINUXWINDOWSAUTO

The target operating system where the script will run.

Default{type: io.kestra.plugin.scripts.runner.docker.Docker, pullPolicy: IF_NOT_PRESENT}

The task runner to use.

Task runners are provided by plugins, each have their own properties.

Definitions

This task runner executes tasks in a container-based Docker-compatible engine. Use the containerImage property to configure the image for the task.

To access the task's working directory, use the {{workingDir}} Pebble expression or the WORKING_DIR environment variable. Input files and namespace files added to the task will be accessible from that directory.

To generate output files, we recommend using the outputFiles task's property. This allows you to explicitly define which files from the task's working directory should be saved as output files.

Alternatively, when writing files in your task, you can leverage the {{outputDir}} Pebble expression or the OUTPUT_DIR environment variable. All files written to that directory will be saved as output files automatically.

Example

Execute a Shell command.

yaml
id: simple_shell_example
namespace: company.team

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
    commands:
    - echo "Hello World"

Pass input files to the task, execute a Shell command, then retrieve output files.

yaml
id: shell_example_with_files
namespace: company.team

inputs:
  - id: file
    type: FILE

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    inputFiles:
      data.txt: "{{ inputs.file }}"
    outputFiles:
      - "*.txt"
    containerImage: centos
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
    commands:
    - cp {{ workingDir }}/data.txt {{ workingDir }}/out.txt

Run a Python script in Docker and allocate a specific amount of memory.

yaml
id: allocate_memory_to_python_script
namespace: company.team

tasks:
  - id: script
    type: io.kestra.plugin.scripts.python.Script
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
      pullPolicy: IF_NOT_PRESENT
      cpu:
        cpus: 1
      memory: 
        memory: "512Mb"
    containerImage: ghcr.io/kestra-io/kestrapy:latest
    script: |
      from kestra import Kestra
      
      data = dict(message="Hello from Kestra!")
      Kestra.outputs(data)
type*Requiredobject
configstringobject

Docker configuration file.

Docker configuration file that can set access credentials to private container registries. Usually located in ~/.docker/config.json.

cpu

Limits the CPU usage to a given maximum threshold value.

By default, each container’s access to the host machine’s CPU cycles is unlimited. You can set various constraints to limit a given container’s access to the host machine’s CPU cycles.

cpusnumberstring

The maximum amount of CPU resources a container can use.

Make sure to set that to a numeric value e.g. cpus: "1.5" or cpus: "4" or For instance, if the host machine has two CPUs and you set cpus: "1.5", the container is guaranteed at most one and a half of the CPUs.

credentials
authstring

The registry authentication.

The auth field is a base64-encoded authentication string of username: password or a token.

identityTokenstring

The identity token.

passwordstring

The registry password.

registrystring

The registry URL.

If not defined, the registry will be extracted from the image name.

registryTokenstring

The registry token.

usernamestring

The registry username.

deletebooleanstring
Defaulttrue

Whether the container should be deleted upon completion.

deviceRequestsarray

A list of device requests to be sent to device drivers.

capabilitiesarray
SubTypearray

A list of capabilities; an OR list of AND lists of capabilities.

countintegerstring
deviceIdsarray
SubTypestring
driverstring
optionsobject
SubTypestring

Driver-specific options, specified as key/value pairs.

These options are passed directly to the driver.

entryPointarray
SubTypestring
Default[ "" ]

Docker entrypoint to use.

extraHostsarray
SubTypestring

Extra hostname mappings to the container network interface configuration.

fileHandlingStrategystring
DefaultVOLUME
Possible Values
MOUNTVOLUME

File handling strategy.

How to handle local files (input files, output files, namespace files, ...). By default, we create a volume and copy the file into the volume bind path. Configuring it to MOUNT will mount the working directory instead.

hoststring

Docker API URI.

killGracePeriodstring
DefaultPT0S
Formatduration

When a task is killed, this property sets the grace period before killing the container.

By default, we kill the container immediately when a task is killed. Optionally, you can configure a grace period so the container is stopped with a grace period instead.

memory

Limits memory usage to a given maximum threshold value.

Docker can enforce hard memory limits, which allow the container to use no more than a given amount of user or system memory, or soft limits, which allow the container to use as much memory as it needs unless certain conditions are met, such as when the kernel detects low memory or contention on the host machine. Some of these options have different effects when used alone or when more than one option is set.

kernelMemorystring

The maximum amount of kernel memory the container can use.

The minimum allowed value is 4MB. Because kernel memory cannot be swapped out, a container which is starved of kernel memory may block host machine resources, which can have side effects on the host machine and on other containers. See the kernel-memory docs for more details.

memorystring

The maximum amount of memory resources the container can use.

Make sure to use the format number + unit (regardless of the case) without any spaces. The unit can be KB (kilobytes), MB (megabytes), GB (gigabytes), etc.

Given that it's case-insensitive, the following values are equivalent:

  • "512MB"
  • "512Mb"
  • "512mb"
  • "512000KB"
  • "0.5GB"

It is recommended that you allocate at least 6MB.

memoryReservationstring

Allows you to specify a soft limit smaller than memory which is activated when Docker detects contention or low memory on the host machine.

If you use memoryReservation, it must be set lower than memory for it to take precedence. Because it is a soft limit, it does not guarantee that the container doesn’t exceed the limit.

memorySwapstring

The total amount of memory and swap that can be used by a container.

If memory and memorySwap are set to the same value, this prevents containers from using any swap. This is because memorySwap includes both the physical memory and swap space, while memory is only the amount of physical memory that can be used.

memorySwappinessstring

A setting which controls the likelihood of the kernel to swap memory pages.

By default, the host kernel can swap out a percentage of anonymous pages used by a container. You can set memorySwappiness to a value between 0 and 100 to tune this percentage.

oomKillDisablebooleanstring

By default, if an out-of-memory (OOM) error occurs, the kernel kills processes in a container.

To change this behavior, use the oomKillDisable option. Only disable the OOM killer on containers where you have also set the memory option. If the memory flag is not set, the host can run out of memory, and the kernel may need to kill the host system’s processes to free the memory.

networkModestring

Docker network mode to use e.g. host, none, etc.

pluginDefaultsRefstring

Reference (ref) of the pluginDefaults to apply to this task runner.

portBindingsarray
SubTypestring

List of port bindings.

Corresponds to the --publish (-p) option of the docker run CLI command using the format ip: dockerHostPort: containerPort/protocol. Possible example :

  • 8080: 80/udp- 127.0.0.1: 8080: 80- 127.0.0.1: 8080: 80/udp
privilegedbooleanstring

Give extended privileges to this container.

pullPolicyobject
resumebooleanstring
Defaulttrue

Whether to resume an existing matching container on restart.

If enabled, the runner will search for an existing container labeled with the current execution/task identifiers and reattach to it instead of creating a new container.

shmSizestring

Size of /dev/shm in bytes.

The size must be greater than 0. If omitted, the system uses 64MB.

userstring

User in the Docker container.

versionstring

Plugin Version

Defines the version of the plugin to use.

The version must follow the Semantic Versioning (SemVer) specification:

  • A single-digit MAJOR version (e.g., 1).
  • A MAJOR.MINOR version (e.g., 1.1).
  • A MAJOR.MINOR.PATCH version, optionally with any qualifier (e.g., 1.1.2, 1.1.0-SNAPSHOT).
volumesarray
SubTypestring

List of volumes to mount.

Make sure to provide a map of a local path to a container path in the format: /home/local/path:/app/container/path. Volume mounts are disabled by default for security reasons — if you are sure you want to use them, enable that feature in the plugin configuration by setting volume-enabled to true.

Here is how you can add that setting to your kestra configuration:

kestra: 
  plugins: 
    configurations: 
      - type: io.kestra.plugin.scripts.runner.docker.Docker
        values: 
          volume-enabled: true
waitbooleanstring
Defaulttrue

Whether to wait for the container to exit.

Executes task commands with the host OS process APIs. Working directory is exposed via {{workingDir}} expression / WORKING_DIR environment variables; if outputFiles are configured, write them to the same directory or use {{outputDir}} / OUTPUT_DIR when enabled. Input files and namespace files will be available in this directory.

Platform-agnostic (Linux/macOS/Windows). If the worker stops, the process is interrupted and will be recreated on restart.

Example

Execute a Shell command.

yaml
id: new_shell
namespace: company.team

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    commands:
      - echo "Hello World"

Install custom Python packages before executing a Python script. Note how we use the --break-system-packages flag to avoid conflicts with the system packages. Make sure to use this flag if you see errors similar to error: externally-managed-environment.

yaml
id: before_commands_example
namespace: company.team

inputs:
  - id: url
    type: URI
    defaults: https://jsonplaceholder.typicode.com/todos/1

tasks:
  - id: transform
    type: io.kestra.plugin.scripts.python.Script
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    beforeCommands:
      - pip install kestra requests --break-system-packages
    script: |
      import requests
      from kestra import Kestra

      url = "{{ inputs.url }}"

      response = requests.get(url)
      print('Status Code:', response.status_code)
      Kestra.outputs(response.json())

Pass input files to the task, execute a Shell command, then retrieve output files.

yaml
id: new_shell_with_file
namespace: company.team

inputs:
  - id: file
    type: FILE

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    inputFiles:
      data.txt: "{{inputs.file}}"
    outputFiles:
      - out.txt
    taskRunner:
      type: io.kestra.plugin.core.runner.Process
    commands:
      - cp {{workingDir}}/data.txt {{workingDir}}/out.txt
type*Requiredobject
pluginDefaultsRefstring

Reference (ref) of the pluginDefaults to apply to this task runner.

versionstring

Plugin Version

Defines the version of the plugin to use.

The version must follow the Semantic Versioning (SemVer) specification:

  • A single-digit MAJOR version (e.g., 1).
  • A MAJOR.MINOR version (e.g., 1.1).
  • A MAJOR.MINOR.PATCH version, optionally with any qualifier (e.g., 1.1.2, 1.1.0-SNAPSHOT).

This plugin is only available in the Enterprise Edition (EE).

This task runner supports ECS (Fargate or EC2) and EKS compute environments.

Make sure to set the containerImage property because this runner runs the task in a container.

To access the task's working directory, use the {{ workingDir }} Pebble expression or the WORKING_DIR environment variable. This directory will contain all input files and namespace files (if enabled).

To generate output files you can either use the outputFiles task property and create a file with the same name in the task's working directory, or create any file in the output directory which can be accessed using the {{ outputDir }} Pebble expression or the OUTPUT_DIR environment variable.

To use inputFiles, outputFiles or namespaceFiles properties, make sure to set the bucket property. The bucket serves as an intermediary storage layer for the task runner. Input and namespace files will be uploaded to the cloud storage bucket before the task run starts. Similarly, the task runner will store outputFiles in this bucket during the task run. In the end, the task runner will make those files available for download and preview from the UI by sending them to internal storage.

The task runner will generate a folder in the configured bucket for each task run. You can access that folder using the {{bucketPath}} Pebble expression or the BUCKET_PATH environment variable.

Note that this task runner executes the task in the root directory. You need to use the {{ workingDir }} Pebble expression or the WORKING_DIR environment variable to access files in the task's working directory.

Note that when the Kestra Worker running this task is terminated, the batch job will still runs until completion, then after restarting, the Worker will resume processing on the existing job unless resume is set to false.

This task runner will return with an exit code according to the following mapping:

  • SUCCEEDED: 0
  • FAILED: 1
  • RUNNING: 2
  • RUNNABLE: 3
  • PENDING: 4
  • STARTING: 5
  • SUBMITTED: 6
  • OTHER: -1

To avoid zombie containers in ECS, you can set the timeout property on the task, and Kestra will terminate the batch job if the task is not completed within the specified duration.

By default, this task runner streams job logs in near real-time via CloudWatch Logs Live Tail, which is priced per GB of log data scanned/streamed and requires long-lived AWS credentials for the duration of the job. For cost-sensitive workloads or when using short-lived/role-chained credentials (which can expire mid-job and interrupt the log stream), set streamLogs to false to disable Live Tail entirely — job polling and exit-code mapping are unaffected either way.

**EKS-specific notes: **

  • CloudWatch logs: Log streaming requires Fluent Bit (or the CloudWatch agent) installed on the EKS cluster and configured to forward pod logs to the /aws/batch/job log group with the job name as the log stream prefix. Without this, no logs will appear in Kestra. This is a cluster-level concern managed by the operator, not by Kestra. A runtime warning is emitted when an EKS compute environment is detected.
  • /bin/sh requirement: When outputFiles, {{ outputDir }}, or syncWorkingDirectory are used, the main container command is wrapped in /bin/sh -c. The container image must include /bin/sh. Most standard images (Ubuntu, Amazon Linux, Alpine) do; distroless or scratch images do not. A runtime warning is emitted as a reminder.
  • Use serviceAccountName with IRSA for IAM authorization instead of taskRoleArn.
  • executionRoleArn and taskRoleArn are ignored for EKS compute environments.
Example

Execute a Shell command in a container on ECS Fargate.

yaml
id: run_container
namespace: company.team

variables:
  region: eu-west-2
  computeEnvironmentArn: "arn:aws:batch:eu-west-2:123456789012:compute-environment/kestraFargateEnvironment"

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.aws.runner.Batch
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
      region: "{{ vars.region }}"
      computeEnvironmentArn: "{{ vars.computeEnvironmentArn }}"
    commands:
      - echo "Hello World"

Pass input files to the task, execute a Shell command, then retrieve the output files.

yaml
id: container_with_input_files
namespace: company.team

inputs:
  - id: file
    type: FILE

variables:
  region: eu-west-2
  computeEnvironmentArn: "arn:aws:batch:eu-central-1:123456789012:compute-environment/kestraFargateEnvironment"

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    inputFiles:
      data.txt: "{{ inputs.file }}"
    outputFiles:
      - out.txt
    containerImage: centos
    taskRunner:
      type: io.kestra.plugin.ee.aws.runner.Batch
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
      region: "{{ vars.region }}"
      bucket: "{{ vars.bucket }}"
      computeEnvironmentArn: "{{ vars.computeEnvironmentArn }}"
    commands:
      - cp {{ workingDir }}/data.txt {{ workingDir }}/out.txt

Run a Python script to fetch environment information on AWS ECS Fargate with AWS Batch

yaml
id: aws_batch_runner
namespace: company.team

variables:
  compute_environment_arn: arn:aws:batch:us-east-1:123456789:compute-environment/kestra
  job_queue_arn: arn:aws:batch:us-east-1:123456789:job-queue/kestra
  execution_role_arn: arn:aws:iam::123456789:role/ecsTaskExecutionRole
  task_role_arn: arn:aws:iam::123456789:role/ecsTaskRole

tasks:
  - id: send_data
    type: io.kestra.plugin.scripts.python.Script
    containerImage: ghcr.io/kestra-io/pydata:latest
    taskRunner:
      type: io.kestra.plugin.ee.aws.runner.Batch
      region: us-east-1
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
      computeEnvironmentArn: "{{ vars.compute_environment_arn }}"
      jobQueueArn: "{{ vars.job_queue_arn }}"
      executionRoleArn: "{{ vars.execution_role_arn }}"
      taskRoleArn: "{{ vars.task_role_arn }}"
      bucket: kestra-us
    script: |
      import platform
      import socket
      import sys

      print("Hello from AWS Batch and kestra!")

      def print_environment_info():
          print(f"Host's network name: {platform.node()}")
          print(f"Python version: {platform.python_version()}")
          print(f"Platform information (instance type): {platform.platform()}")
          print(f"OS/Arch: {sys.platform}/{platform.machine()}")

          try:
              hostname = socket.gethostname()
              ip_address = socket.gethostbyname(hostname)
              print(f"Host IP Address: {ip_address}")
          except socket.error as e:
              print("Unable to obtain IP address.")

      if __name__ == '__main__':
          print_environment_info()

Execute a Shell command on AWS Batch with an EKS compute environment.

yaml
id: run_container_on_eks
namespace: company.team

variables:
  region: us-east-1
  compute_environment_arn: arn:aws:batch:us-east-1:123456789:compute-environment/kestraEksEnvironment
  job_queue_arn: arn:aws:batch:us-east-1:123456789:job-queue/kestraEksQueue

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    containerImage: amazonlinux:2
    taskRunner:
      type: io.kestra.plugin.ee.aws.runner.Batch
      region: "{{ vars.region }}"
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
      computeEnvironmentArn: "{{ vars.compute_environment_arn }}"
      jobQueueArn: "{{ vars.job_queue_arn }}"
      serviceAccountName: kestra-sa
    commands:
      - echo "Hello from AWS Batch on EKS"
computeEnvironmentArn*Requiredstring

Compute environment in which to run the job

region*Requiredstring

AWS region with which the SDK should communicate

type*Requiredobject
accessKeyIdstring

Access Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

bucketstring

S3 Bucket to upload (inputFiles and namespaceFiles) and download (outputFiles) files

It's mandatory to provide a bucket if you want to use such properties.

completionCheckIntervalstring
DefaultPT5S

Determines how often Kestra should poll the container for completion. By default, the task runner checks every 5 seconds whether the job is completed. You can set this to a lower value (e.g. PT0.1S = every 100 milliseconds) for quick jobs and to a lower threshold (e.g. PT1M = every minute) for long-running jobs. Setting this property to a lower value will reduce the number of API calls Kestra makes to the remote service — keep that in mind in case you see API rate limit errors

deletebooleanstring
Defaulttrue

Whether the job should be deleted upon completion

Warning, if the job is not deleted, a retry of the task could resume an old failed attempt of the job.

endpointOverridestring

The endpoint with which the SDK should communicate

This property allows you to use a different S3 compatible storage backend.

executionRoleArnstring

Execution role for the AWS Batch job

Mandatory if the compute environment is ECS Fargate. See the AWS documentation for more details. Ignored for EKS compute environments.

jobQueueArnstring

Job queue to use to submit jobs (ARN). If not specified, the task runner will create a job queue — keep in mind that this can lead to a longer execution

pluginDefaultsRefstring

Reference (ref) of the pluginDefaults to apply to this task runner.

resources
Default{ "request": { "memory": "2048", "cpu": "1" } }

Custom resources for the container

For ECS Fargate, see the AWS documentation for supported CPU/memory combinations. For EKS, values follow Kubernetes resource format: CPU as a decimal (e.g. 0.5, 1) and memory as an integer in MiB (e.g. 2048).

request*Required
cpu*Requiredstring
memory*Requiredstring
resumebooleanstring
Defaulttrue

Whether to reconnect to the current job if it already exists

secretKeyIdstring

Secret Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

serviceAccountNamestring

Kubernetes service account name for the EKS pod

Used with IRSA (IAM Roles for Service Accounts) for IAM authorization in EKS compute environments. Ignored for ECS.

sessionTokenstring

AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

sidecarResources

Resources for sidecar containers

Resources used by the internal inputFiles and outputFiles containers (S3 upload/download). If not set, defaults to minimal valid values depending on the compute environment (Fargate: 0.25 vCPU / 512 MiB; EC2: 1 vCPU / 128 MiB). Note: on Fargate, AWS Batch validates resources at the task level; sidecar resources are therefore subtracted from the main container to keep the overall task resources equal to resources.request.

request*Required
cpu*Requiredstring
memory*Requiredstring
streamLogsbooleanstring
Defaulttrue

Whether to stream job logs in near real-time using CloudWatch Logs Live Tail

When true (default), the task runner starts a CloudWatch Logs Live Tail session so job logs appear in the Kestra execution as they are produced. Live Tail is priced per GB of log data scanned/streamed and requires the AWS credentials to remain valid for the entire job duration; if short-lived or role-chained credentials expire mid-job, the log stream is interrupted and the task fails even if the underlying AWS Batch job succeeds. Set this to false to skip Live Tail entirely — no CloudWatch API calls are made — job polling and exit-code mapping are unaffected. Job container logs are still written to the /aws/batch/job CloudWatch log group by the awslogs log driver (ECS) or Fluent Bit (EKS) regardless of this setting; only near-real-time streaming into Kestra is disabled. Note: unlike the unrelated streamLogs property on io.kestra.plugin.ee.aws.batch.Run (which defaults to false), this defaults to true because Live Tail is this task runner's only way to surface job output in the Kestra execution as it happens.

stsEndpointOverridestring

The AWS STS endpoint with which the SDKClient should communicate

stsRoleArnstring

AWS STS Role

The Amazon Resource Name (ARN) of the role to assume. If set the task will use the StsAssumeRoleCredentialsProvider. If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

stsRoleExternalIdstring

AWS STS External Id

A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.

stsRoleSessionDurationstring
DefaultPT15M

AWS STS Session duration

The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.

stsRoleSessionNamestring

AWS STS Session name

This property is only used when an stsRoleArn is defined.

syncWorkingDirectorybooleanstring
Defaultfalse

Whether to synchronize working directory from remote runner back to local one after run.

taskRoleArnstring

Task role to use within the container

Needed if you want to authenticate with AWS CLI within your container. Ignored for EKS compute environments.

versionstring

Plugin Version

Defines the version of the plugin to use.

The version must follow the Semantic Versioning (SemVer) specification:

  • A single-digit MAJOR version (e.g., 1).
  • A MAJOR.MINOR version (e.g., 1.1).
  • A MAJOR.MINOR.PATCH version, optionally with any qualifier (e.g., 1.1.2, 1.1.0-SNAPSHOT).
waitUntilCompletionstring
DefaultPT1H

The maximum duration to wait for the job completion unless the task timeout property is set which will take precedence over this property

AWS Batch will automatically timeout the job upon reaching that duration and the task will be marked as failed.

workingDirectorySizeLimitintegerstring
Default20480

Maximum storage in MiB for the task's working directory on EKS compute environments

Size limit of the working-directory volume backing the EKS pod, in mebibytes (MiB). Raise it for tasks that sync large input or output files. Ignored for ECS compute environments.

Enterprise Edition only.

This task runner launches an EC2 instance, runs your task's commands directly on it via AWS Systems Manager Run Command (no SSH, no container), then terminates the instance. Reach for it when your work must run natively on a machine, such as native binaries or licensed software tied to a specific AMI. The containerImage property is ignored.

Requirements: the AMI must have the SSM Agent installed (Amazon Linux and recent Ubuntu AMIs have it), and iamInstanceProfile must grant SSM access. To pass inputFiles, outputFiles, or namespaceFiles, set the bucket property, which the runner uses to stage files. Access the working directory with {{ workingDir }} or the WORKING_DIR environment variable.

For a completed command, the task's exit code is the real exit code of your last command. Other terminal SSM statuses (e.g. cancelled or timed out) map to a fixed code instead. Command output appears in the Kestra logs. If the Worker restarts mid-run, resume (default true) reattaches to the same instance instead of starting a new one. The instance is always terminated when the run ends, whether it succeeds, fails, or is killed.

See the plugin documentation for the full IAM, networking, and logging setup.

Example

Launch an EC2 instance and run a Shell command on it.

yaml
id: ec2_shell
namespace: company.team

variables:
  region: eu-west-3
  amiId: ami-0c55b159cbfafe1f0

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.aws.runner.Ec2
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
      region: "{{ vars.region }}"
      amiId: "{{ vars.amiId }}"
      instanceType: t3.micro
      iamInstanceProfile: kestra-ec2-ssm-profile
    commands:
      - echo "Hello World"

Pass input files to the task, execute a Shell command, then retrieve the output files.

yaml
id: ec2_with_input_files
namespace: company.team

inputs:
  - id: file
    type: FILE

variables:
  region: eu-west-3
  amiId: ami-0c55b159cbfafe1f0
  bucket: my-kestra-staging-bucket

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    inputFiles:
      data.txt: "{{ inputs.file }}"
    outputFiles:
      - out.txt
    taskRunner:
      type: io.kestra.plugin.ee.aws.runner.Ec2
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
      region: "{{ vars.region }}"
      amiId: "{{ vars.amiId }}"
      instanceType: t3.micro
      iamInstanceProfile: kestra-ec2-ssm-profile
      subnetId: subnet-0123456789abcdef0
      securityGroupIds:
        - sg-0123456789abcdef0
      bucket: "{{ vars.bucket }}"
    commands:
      - cp {{ workingDir }}/data.txt {{ workingDir }}/out.txt
amiId*Requiredstring

AMI ID to launch the instance from

The ID of the Amazon Machine Image used to launch the instance, for example ami-0c55b159cbfafe1f0. The AMI must have the SSM Agent installed and running (true by default for modern Amazon Linux 2/2023 and Ubuntu AMIs), since this task runner executes the task via AWS Systems Manager Run Command rather than SSH.

iamInstanceProfile*Requiredstring

IAM instance profile

Name or ARN of the IAM instance profile attached to the instance. Must grant at minimum the AmazonSSMManagedInstanceCore managed policy (or equivalent) so the SSM Agent can register with Systems Manager and receive commands, plus S3 read/write access to bucket if file staging is used. AmazonSSMManagedInstanceCore alone does not grant CloudWatch Logs permissions: if streamLogs is true (the default), also grant logs: CreateLogGroup, logs: CreateLogStream, and logs: PutLogEvents.

instanceType*Requiredstring

EC2 instance type

The compute capacity of the instance, for example t3.micro or m5.large. See the AWS documentation for available types.

region*Requiredstring

AWS region with which the SDK should communicate

type*Requiredobject
accessKeyIdstring

Access Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

bucketstring

S3 Bucket to upload (inputFiles and namespaceFiles) and download (outputFiles) files

It's mandatory to provide a bucket if you want to use such properties.

completionCheckIntervalstring
DefaultPT5S

Determines how often Kestra should poll the SSM command for completion. By default, the task runner checks every 5 seconds whether the command is completed. You can set this to a lower value (e.g. PT0.1S = every 100 milliseconds) for quick tasks and to a higher threshold (e.g. PT1M = every minute) for long-running tasks. Setting this property to a lower value will reduce the number of API calls Kestra makes to the remote service, so keep that in mind in case you see API rate limit errors

endpointOverridestring

The endpoint with which the SDK should communicate

This property allows you to use a different S3 compatible storage backend.

instanceReadyTimeoutstring
DefaultPT5M

Instance and SSM Agent readiness timeout

The maximum duration to wait for the launched instance to reach the running state and for its SSM Agent to register with Systems Manager, before failing the task. Defaults to PT5M. Heavy or custom AMIs (e.g. ones that run lengthy boot-time provisioning) may need a higher value.

pluginDefaultsRefstring

Reference (ref) of the pluginDefaults to apply to this task runner.

resumebooleanstring
Defaulttrue

Whether to reconnect to the current instance and command if they already exist

secretKeyIdstring

Secret Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

securityGroupIdsarray
SubTypestring

Security groups to attach to the instance

List of security group IDs to associate with the instance's network interface. If not set, the subnet's default security group is used.

sessionTokenstring

AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

spotMaxPricestring

Maximum Spot price per hour

If set, the instance is launched as a Spot Instance with this maximum hourly price in USD (e.g. 0.05). Leave unset to launch a standard On-Demand instance. Spot instances can be reclaimed by AWS at any time. Only use this for interruption-tolerant workloads.

streamLogsbooleanstring
Defaulttrue

Whether to poll CloudWatch Logs for near real-time command output

When true (default), the task runner polls the /aws/ec2/kestra-run-command CloudWatch log group for this command's output while it runs, so it appears in the Kestra execution as it is produced. This requires the instance's iamInstanceProfile to grant logs: CreateLogGroup, logs: CreateLogStream, and logs: PutLogEvents (AmazonSSMManagedInstanceCore alone does not include these). Set this to false to skip that live polling entirely: no CloudWatch API calls are made. Either way, command output always reaches Kestra once the command finishes: the task runner falls back to reading StandardOutputContent/StandardErrorContent straight from SSM's GetCommandInvocation response (capped at roughly 24 KB per stream) whenever nothing was already streamed from CloudWatch. Command polling and exit-code mapping are unaffected by this property either way.

stsEndpointOverridestring

The AWS STS endpoint with which the SDKClient should communicate

stsRoleArnstring

AWS STS Role

The Amazon Resource Name (ARN) of the role to assume. If set the task will use the StsAssumeRoleCredentialsProvider. If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

stsRoleExternalIdstring

AWS STS External Id

A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.

stsRoleSessionDurationstring
DefaultPT15M

AWS STS Session duration

The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.

stsRoleSessionNamestring

AWS STS Session name

This property is only used when an stsRoleArn is defined.

subnetIdstring

Subnet to launch the instance in

The ID of the subnet the instance's network interface is attached to. If not set, AWS launches the instance into your account's default VPC/subnet. The instance must be able to reach the AWS Systems Manager and (if bucket is set) S3 endpoints, either via a NAT/internet gateway or VPC endpoints.

syncWorkingDirectorybooleanstring
Defaultfalse

Whether to synchronize working directory from remote runner back to local one after run.

versionstring

Plugin Version

Defines the version of the plugin to use.

The version must follow the Semantic Versioning (SemVer) specification:

  • A single-digit MAJOR version (e.g., 1).
  • A MAJOR.MINOR version (e.g., 1.1).
  • A MAJOR.MINOR.PATCH version, optionally with any qualifier (e.g., 1.1.2, 1.1.0-SNAPSHOT).
waitForLogIntervalstring
DefaultPT5S

Post-completion CloudWatch log drain period

Once the command reaches a terminal status, the task runner keeps polling CloudWatch Logs until no new output has arrived for this long, so a short command's entire output burst, which can land in CloudWatch a second or two after the command already finished, isn't lost. Only used when streamLogs is true. Defaults to PT5S.

waitUntilCompletionstring
DefaultPT1H

The maximum duration to wait for the SSM command completion unless the task timeout property is set which will take precedence over this property

The task will be marked as failed if the command has not completed by the time this duration elapses.

This plugin is only available in the Enterprise Edition (EE).

This task runner is container-based so the containerImage property must be set.

To access the task's working directory, use the {{workingDir}} Pebble expression or the WORKING_DIR environment variable. Input files and namespace files will be available in this directory.

To generate output files you can either use the outputFiles task's property and create a file with the same name in the task's working directory, or create any file in the output directory which can be accessed by the {{outputDir}} Pebble expression or the OUTPUT_DIR environment variables.

To use inputFiles, outputFiles or namespaceFiles properties, make sure to set the blobStorage property. The blob storage serves as an intermediary storage layer for the task runner. Input and namespace files will be uploaded to the cloud storage bucket before the task run. Similarly, the task runner will store outputFiles in this blob storage during the task run. In the end, the task runner will make those files available for download and preview from the UI by sending them to internal storage.

The task runner will generate a folder in the configured blobStorage for each task run. You can access that folder using the {{bucketPath}} Pebble expression or the BUCKET_PATH environment variable. There is two supported way to provide authentication for the blob storage:

  • connectionString and containerName properties
  • containerName, endpoint, sharedKeyAccountName and sharedKeyAccountAccessKey properties

Note that when the Kestra Worker running this task is terminated, the batch job will still runs until completion, then after restarting, the Worker will resume processing on the existing job unless resume is set to false.

Example

Execute a Shell command.

yaml
id: new_shell
namespace: company.team

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.azure.runner.Batch
      account: "{{secrets.account}}"
      accessKey: "{{secrets.accessKey}}"
      endpoint: "{{secrets.endpoint}}"
      poolId: "{{vars.poolId}}"
    commands:
      - echo "Hello World"

Pass input files to the task, execute a Shell command, then retrieve output files.

yaml
id: new_shell_with_file
namespace: company.team

inputs:
  - id: file
    type: FILE

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    inputFiles:
      data.txt: "{{inputs.file}}"
    outputFiles:
      - out.txt
    containerImage: centos
    taskRunner:
      type: io.kestra.plugin.ee.azure.runner.Batch
      account: "{{secrets.account}}"
      accessKey: "{{secrets.accessKey}}"
      endpoint: "{{secrets.endpoint}}"
      poolId: "{{vars.poolId}}"
      blobStorage:
        connectionString: "{{secrets.connectionString}}"
        containerName: "{{vars.containerName}}"
    commands:
      - cp {{workingDir}}/data.txt {{workingDir}}/out.txt

Run a Python script to fetch environment information on Azure with Azure Batch VMs

yaml
id: azure_batch_runner
namespace: company.team

variables:
  pool_id: poolId
  container_name: containerName

tasks:
  - id: scrape_environment_info
    type: io.kestra.plugin.scripts.python.Commands
    containerImage: ghcr.io/kestra-io/pydata:latest
    taskRunner:
      type: io.kestra.plugin.ee.azure.runner.Batch
      account: "{{ secret('AZURE_ACCOUNT') }}"
      accessKey: "{{ secret('AZURE_ACCESS_KEY') }}"
      endpoint: "{{ secret('AZURE_ENDPOINT') }}"
      poolId: "{{ vars.pool_id }}"
      blobStorage:
        containerName: "{{ vars.container_name }}"
        connectionString: "{{ secret('AZURE_CONNECTION_STRING') }}"
    commands:
      - python {{ workingDir }}/main.py
    namespaceFiles:
      enabled: true
    outputFiles:
      - environment_info.json
    inputFiles:
      main.py: |
        import platform
        import socket
        import sys
        import json

        from kestra import Kestra

        print("Hello from Azure Batch and kestra!")

        def print_environment_info():
            print(f"Host's network name: {platform.node()}")
            print(f"Python version: {platform.python_version()}")
            print(f"Platform information (instance type): {platform.platform()}")
            print(f"OS/Arch: {sys.platform}/{platform.machine()}")

            env_info = {
                "host": platform.node(),
                "platform": platform.platform(),
                "OS": sys.platform,
                "python_version": platform.python_version(),
            }
            Kestra.outputs(env_info)

            filename = 'environment_info.json'
            with open(filename, 'w') as json_file:
                json.dump(env_info, json_file, indent=4)

        if __name__ == '__main__':
          print_environment_info()
accessKey*Requiredstring

Batch account access key

account*Requiredstring

Batch account name

endpoint*Requiredstring

The blob service endpoint.

poolId*Requiredstring

Id of the pool on which to run the job

type*Requiredobject
Possible Values
io.kestra.plugin.ee.azure.runner.Batchio.kestra.plugin.azure.runner.Batch
blobStorage

Blob storage configuration for staging batch task files

containerName*Requiredstring

The URL of the blob container the compute node should use.

Mandatory if you want to use namespaceFiles, inputFiles or outputFiles properties.

connectionStringstring

Connection string of the Storage Account.

endpointstring

The blob service endpoint.

sharedKeyAccountAccessKeystring

Shared Key access key for authenticating requests.

sharedKeyAccountNamestring

Shared Key account name for authenticating requests.

completionCheckIntervalstring
DefaultPT5S

Determines how often Kestra should poll the container for completion. By default, the task runner checks every 5 seconds whether the job is completed. You can set this to a lower value (e.g. PT0.1S = every 100 milliseconds) for quick jobs and to a lower threshold (e.g. PT1M = every minute) for long-running jobs. Setting this property to a lower value will reduce the number of API calls Kestra makes to the remote service — keep that in mind in case you see API rate limit errors

deletebooleanstring
Defaulttrue

Whether the job should be deleted upon completion

Warning, if the job is not deleted, a retry of the task could resume an old failed attempt of the job.

pluginDefaultsRefstring

Reference (ref) of the pluginDefaults to apply to this task runner.

registry

The private registry which contains the container image

identityReference

User-assigned identity for ACR

Use managed identity instead of username/password when supported by the registry

resourceIdstring

User-assigned identity resource ID

ARM resource ID of the managed identity used for storage access

passwordstring

Registry password

registryServerstring

Registry server

Container registry hostname; defaults to docker.io

userNamestring

Registry username

resumebooleanstring
Defaulttrue

Whether to reconnect to the current job if it already exists

streamLogsbooleanstring
Defaultfalse

Enable log streaming during task execution

This property is useful for capturing logs from tasks that have a timeout. If a task with a timeout is terminated, this property makes sure all logs up to that point are retrieved.

syncWorkingDirectorybooleanstring
Defaultfalse

Whether to synchronize working directory from remote runner back to local one after run.

versionstring

Plugin Version

Defines the version of the plugin to use.

The version must follow the Semantic Versioning (SemVer) specification:

  • A single-digit MAJOR version (e.g., 1).
  • A MAJOR.MINOR version (e.g., 1.1).
  • A MAJOR.MINOR.PATCH version, optionally with any qualifier (e.g., 1.1.2, 1.1.0-SNAPSHOT).
waitUntilCompletionstring
DefaultPT1H

The maximum duration to wait for the job completion unless the task timeout property is set, which will take precedence over this property

Azure Batch will automatically timeout the job upon reaching such duration and the task will fail.

Enterprise Edition only.

Runs the task's commands on an Azure VM via the Azure Run Command API: no SSH, no container. containerImage is ignored.

The VM has no public IP and no inbound rules: Run Command reaches it through the Azure control plane.

To create a VM, set subnetId to an existing subnet. The runner never creates a virtual network.

vmName targets an existing VM, which is never stopped or deleted. stopVm/deleteVm (both default true) apply only to VMs the runner created, and are honored on success, failure, or kill. Deleting a created VM also removes its network interface and OS disk.

Staging inputFiles, outputFiles, namespaceFiles, or {{ outputDir }} needs blobStorage: the Worker stages files via Blob Storage and gives the VM a short-lived, container-scoped SAS token used with plain curl (no Azure CLI assumed). The subnet needs outbound access to Blob Storage or staging fails.

Run Command is synchronous with no reattachable command ID, so on a Worker restart resume re-issues the command against the same VM: commands should tolerate running more than once.

Command output is only returned once the command completes, there is no live log streaming, and Azure truncates captured stdout/stderr to about 4 KB per stream, so a very verbose task may lose part of its output.

Example

Run a Python script on a new Azure VM.

yaml
id: azure_vm_python_task
namespace: company.team

tasks:
  - id: run_python
    type: io.kestra.plugin.scripts.python.Script
    taskRunner:
      type: io.kestra.plugin.ee.azure.runner.VirtualMachine
      tenantId: "{{ secret('AZURE_TENANT_ID') }}"
      clientId: "{{ secret('AZURE_CLIENT_ID') }}"
      clientSecret: "{{ secret('AZURE_CLIENT_SECRET') }}"
      subscriptionId: "{{ secret('AZURE_SUBSCRIPTION_ID') }}"
      resourceGroupName: kestra-runners
      region: eastus
      vmSize: Standard_DS2_v2
      subnetId: "/subscriptions/{{ secret('AZURE_SUBSCRIPTION_ID') }}/resourceGroups/kestra-runners/providers/Microsoft.Network/virtualNetworks/kestra-vnet/subnets/kestra-subnet"
    script: |
      print("Hello from Azure VM!")

Reuse an existing VM and keep it running after the task completes.

yaml
id: azure_vm_reuse
namespace: company.team

tasks:
  - id: process
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.azure.runner.VirtualMachine
      tenantId: "{{ secret('AZURE_TENANT_ID') }}"
      clientId: "{{ secret('AZURE_CLIENT_ID') }}"
      clientSecret: "{{ secret('AZURE_CLIENT_SECRET') }}"
      subscriptionId: "{{ secret('AZURE_SUBSCRIPTION_ID') }}"
      resourceGroupName: kestra-runners
      vmName: my-existing-vm
      stopVm: false
      deleteVm: false
    commands:
      - echo "Processing on reused VM"

Pass input files to the task, execute a Shell command, then retrieve the output files.

yaml
id: azure_vm_with_input_files
namespace: company.team

inputs:
  - id: file
    type: FILE

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    inputFiles:
      data.txt: "{{ inputs.file }}"
    outputFiles:
      - out.txt
    taskRunner:
      type: io.kestra.plugin.ee.azure.runner.VirtualMachine
      tenantId: "{{ secret('AZURE_TENANT_ID') }}"
      clientId: "{{ secret('AZURE_CLIENT_ID') }}"
      clientSecret: "{{ secret('AZURE_CLIENT_SECRET') }}"
      subscriptionId: "{{ secret('AZURE_SUBSCRIPTION_ID') }}"
      resourceGroupName: kestra-runners
      region: eastus
      vmSize: Standard_DS1_v2
      subnetId: "/subscriptions/{{ secret('AZURE_SUBSCRIPTION_ID') }}/resourceGroups/kestra-runners/providers/Microsoft.Network/virtualNetworks/kestra-vnet/subnets/kestra-subnet"
      blobStorage:
        connectionString: "{{ secret('AZURE_CONNECTION_STRING') }}"
        containerName: kestra-vm-staging
    commands:
      - cp {{ workingDir }}/data.txt {{ workingDir }}/out.txt
resourceGroupName*Requiredstring

Resource group containing the virtual machine

Used both to look up an existing VM (vmName) and to create a new one.

subscriptionId*Requiredstring

Azure subscription ID

The subscription that owns resourceGroupName and the virtual machine.

type*Requiredobject
blobStorage

Blob storage used to stage inputFiles, outputFiles, namespaceFiles, and {{ outputDir }}

Mandatory to set if you want to use these properties.

containerName*Requiredstring

The URL of the blob container the compute node should use.

Mandatory if you want to use namespaceFiles, inputFiles or outputFiles properties.

connectionStringstring

Connection string of the Storage Account.

endpointstring

The blob service endpoint.

sharedKeyAccountAccessKeystring

Shared Key access key for authenticating requests.

sharedKeyAccountNamestring

Shared Key account name for authenticating requests.

clientIdstring

Service principal client (application) ID

Required together with tenantId and clientSecret for service-principal authentication.

clientSecretstring

Service principal client secret

Required together with tenantId and clientId for service-principal authentication.

completionCheckIntervalstring
DefaultPT5S

Determines how often Kestra polls while waiting for the VM to become ready and for the Run Command to complete

By default, every 5 seconds. Lower it for quick tasks, raise it (e.g. PT1M) for long-running ones to reduce API calls.

deleteVmbooleanstring
Defaulttrue

Whether to delete the VM once the task finishes

Defaults to true. Set to false (together with stopVm: false) to keep a shared/reusable VM alive across runs.

imagestring
DefaultCanonical:0001-com-ubuntu-server-jammy:22_04-lts-gen2

Linux image to provision the VM from

Format publisher: offer: sku; the latest version of that image is used. Defaults to Ubuntu 22.04 LTS. Ignored when reusing an existing VM.

instanceReadyTimeoutstring
DefaultPT5M

VM readiness timeout

The maximum duration to wait for a newly created or restarted VM to reach the running power state before failing the task. Defaults to PT5M.

pluginDefaultsRefstring

Reference (ref) of the pluginDefaults to apply to this task runner.

regionstring

Azure region to create the virtual machine in

For example eastus. Required unless vmName refers to an already existing VM.

resumebooleanstring
Defaulttrue

Whether to reconnect to an already running Kestra-created VM instead of creating a new one

Only applies when vmName is not set: on a Worker restart mid-run, the task runner looks for a VM it previously created for this same task run (matched by tags) and reuses it rather than launching a duplicate. The command itself is always re-issued against the resumed VM regardless (see the class description).

stopVmbooleanstring
Defaulttrue

Whether to stop (deallocate) the VM once the task finishes

Deallocating stops compute billing (a plain "power off" alone does not, since Azure still reserves the compute capacity). Defaults to true. Honored whether the task succeeds, fails, or is killed.

subnetIdstring

Subnet the virtual machine's network interface joins

Full resource ID of an existing subnet, for example /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Network/virtualNetworks/<vnet>/subnets/<subnet>. Required to create a new VM, ignored when reusing one via vmName. The runner never creates a virtual network, so the subnet must already provide any outbound network path the VM needs (see blobStorage).

syncWorkingDirectorybooleanstring
Defaultfalse

Whether to download the whole working directory back from Blob Storage instead of just the declared outputFiles

Defaults to false. Useful when the set of files produced by the task isn't known in advance.

tenantIdstring

Azure Active Directory tenant ID

Required together with clientId and clientSecret for service-principal authentication. When all three are left unset, the task runner falls back to DefaultAzureCredential (e.g. a managed identity attached to the Kestra Worker).

versionstring

Plugin Version

Defines the version of the plugin to use.

The version must follow the Semantic Versioning (SemVer) specification:

  • A single-digit MAJOR version (e.g., 1).
  • A MAJOR.MINOR version (e.g., 1.1).
  • A MAJOR.MINOR.PATCH version, optionally with any qualifier (e.g., 1.1.2, 1.1.0-SNAPSHOT).
vmNamestring

Name of an already existing virtual machine to run the task on

Must reference a VM that already exists in resourceGroupName. The runner never creates a VM when vmName is set, and fails if none is found. Since it didn't create the VM, stopVm and deleteVm are ignored for it. Leave unset to have the runner provision a new VM, or resume one it created earlier.

vmSizestring

Virtual machine size

For example Standard_DS1_v2. See the Azure documentation for available sizes. Required unless vmName refers to an already existing VM.

waitUntilCompletionstring
DefaultPT1H

The maximum duration to wait for the Run Command completion unless the task timeout property is set which will take precedence over this property

The task will be marked as failed if the command has not completed by the time this duration elapses.

This plugin is only available in the Enterprise Edition (EE).

To generate output files you can:

  • Use the outputFiles property of the task and create a file with the same name in the task’s working directory, or
  • Create any file in the output directory, which can be accessed with the {{outputDir}} Pebble expression or the OUTPUT_DIR environment variable.

When the Kestra Worker running this task is terminated, the pod continues until completion. After restarting, the Worker will resume processing on the existing pod unless resume is set to false.

If your cluster is configured with RBAC, the service account running your pod must have the following authorizations:

  • pods: get, create, delete, watch, list
  • pods/log: get, watch
  • pods/exec: get, watch
  • secrets: create, delete (only required when credentials is set, to manage the ephemeral imagePullSecret)

Here is an example role that grants these authorizations:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: 
  name: task-runner
rules: 
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "create", "delete", "watch", "list"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["get", "watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "watch"]
Example

Execute a Shell command.

yaml
id: new-shell
namespace: company.team

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.kubernetes.runner.Kubernetes
    commands:
      - echo "Hello World"

Pass input files to the task, execute a Shell command, then retrieve output files.

yaml
id: new-shell-with-file
namespace: company.team

inputs:
  - id: file
    type: FILE

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    inputFiles:
      data.txt: "{{ inputs.file }}"
    outputFiles:
      - out.txt
    containerImage: centos
    taskRunner:
      type: io.kestra.plugin.ee.kubernetes.runner.Kubernetes
    commands:
      - cp {{ workingDir }}/data.txt {{ workingDir }}/out.txt
pullPolicy*Requiredobject
type*Requiredobject
Possible Values
io.kestra.plugin.ee.kubernetes.runner.Kubernetesio.kestra.plugin.kubernetes.runner.Kubernetes
config

Configure Kubernetes cluster connection

Cluster endpoint, auth, and TLS settings used by the runner.

apiVersionstring
Defaultv1

API version

API group version used by the client. Default v1.

caCertDatastring

CA certificate data

Base64-encoded PEM CA bundle. Whitespace is stripped automatically.

caCertFilestring

CA certificate file

Path to a PEM CA bundle.

clientCertDatastring

Client certificate data

Base64-encoded client cert. Whitespace is stripped automatically.

clientCertFilestring

Client certificate file

clientKeyAlgostring
DefaultRSA

Client key algorithm

Algorithm for the client key. Default RSA.

clientKeyDatastring

Client key data

Base64-encoded client key. Whitespace is stripped automatically.

clientKeyFilestring

Client key file

clientKeyPassphrasestring

Client key passphrase

disableHostnameVerificationbooleanstring

Disable hostname verification

Disables TLS hostname checks. Avoid in production clusters.

enableHttp2booleanstring

Enable HTTP/2

Off by default (HTTP/1.1, the prior behavior). Enable to use HTTP/2, which keeps long-lived watch/log streams alive better through proxies like GKE konnectivity.

keyStoreFilestring

Keystore file

keyStorePassphrasestring

Keystore passphrase

masterUrlstring
Defaulthttps://kubernetes.default.svc

Kubernetes API URL

API server endpoint. Default https://kubernetes.default.svc.

maxConcurrentRequestsintegerstring

Max concurrent HTTP requests

Caps total concurrent HTTP requests issued by this client. Lower this when running many concurrent tasks against a rate-limited API server (e.g. GKE). Fabric8 default: 64.

maxConcurrentRequestsPerHostintegerstring

Max concurrent HTTP requests per host

Caps concurrent HTTP requests to a single host (the API server). Fabric8 default: 5.

namespacestring

Default namespace

Namespace used when resources omit a namespace.

oauthTokenstring

OAuth token

oauthTokenProvider

Dynamically refreshes the OAuth token used to authenticate with the Kubernetes API. Use this when authenticating via a cloud provider (e.g. GCP, AWS, Azure) whose tokens are short-lived (typically 1 hour). Without this, long-running tasks will fail with an unauthorized error once the initial token expires. The provider executes a Kestra task to fetch a fresh token each time the Kubernetes client needs to re-authenticate.

Dynamically refreshes an OAuth token by executing a Kestra task each time the Kubernetes client needs to authenticate. This is essential for cloud providers (e.g. GCP, AWS, Azure) whose tokens are short-lived (typically 1 hour). Without automatic refresh, long-running tasks will fail with an unauthorized error once the initial token expires.

cachestring
DefaultPT5M
Formatduration

Token cache duration

How long a fetched token is reused before the provider executes the task again. Caching avoids hammering the token endpoint on every Kubernetes API call. Defaults to 5 minutes (PT5M). Set to PT0S to disable caching.

outputstring

Token output expression

An expression to extract the token value from the task output, e.g. {{ accessToken.tokenValue }}.

task

Token provider task

A Kestra task that fetches a fresh OAuth token. This task is executed each time the Kubernetes client needs to authenticate, enabling automatic token refresh for long-running operations.

passwordstring

Password

trustCertsbooleanstring

Trust all certificates

When true, skips TLS cert validation. Use only for testing.

trustStoreFilestring

Truststore file

trustStorePassphrasestring

Truststore passphrase

usernamestring

Username

watchReconnectIntervalstring

Watch reconnect interval

Backoff between watch reconnect attempts. Increase to prevent reconnect storms under API server pressure. Fabric8 default: 1s.

websocketPingIntervalstring

WebSocket ping interval

Keepalive ping on exec/log connections so idle streams aren't dropped by a proxy (e.g. GKE konnectivity). Only honored by the OkHttp backend. 0 disables.

containerDefaultSpecobject

Default container spec applied to all containers

Merged into every container in the pod: the Kestra main container, user-defined sidecars from podSpec.containers, and Kestra's file-transfer init/sidecar containers.

This is the correct place for settings that must be uniform across all containers:

  • volumeMounts — declare a shared volume mount once instead of repeating it per container
  • securityContext — enforce a consistent security posture across all containers
  • resources — set default limits/requests that containers can override individually
  • env — inject common environment variables into all containers

Nested objects (securityContext, resources) are deep-merged; container-specific values win. Lists (volumeMounts, env) are prepended from the default; for env vars the last entry with a given name wins (container-specific overrides the default). To apply settings only to Kestra's file-transfer containers, use fileSideCarSpec instead.

Example — mounting a shared Docker socket into all containers:

containerDefaultSpec: 
  volumeMounts: 
    - name: docker-socket
      mountPath: /dind
containerSpecobject

Augment main container spec

Merged into the Kestra-generated main container before defaults; supports template expressions.

Environment variables defined here are preserved and merged with Kestra-injected vars (e.g. WORKING_DIR, OUTPUT_DIR). When a variable name collides, the Kestra-injected value takes precedence.

credentials

When set, the runner creates an ephemeral kubernetes.io/dockerconfigjson imagePullSecret in the pod namespace, references it from the task pod, and deletes it together with the pod. Field names mirror the Docker task runner's credentials block, so a flow can switch runner type without changing its credentials definition. Requires the runner ServiceAccount to be allowed to create/delete secrets in the namespace.

Turned into an ephemeral kubernetes.io/dockerconfigjson imagePullSecret created in the pod namespace, referenced by the task pod, and deleted with the pod. The runner ServiceAccount must be allowed to create and delete secrets in that namespace.

authstring

The registry authentication

A base64-encoded username: password string. When set, it is used as-is; otherwise it is computed from username and password.

passwordstring

The registry password

registrystring

The registry URL

If not defined, the registry host is extracted from the image name.

usernamestring

The registry username

deletebooleanstring
Defaulttrue

Delete pod after completion

Defaults to true; set false to keep the pod for debugging.

fileSideCarSpecobject

Augment file sidecar spec

Merged into init/sidecar containers used for file transfer; supports template expressions.

fileSidecar
Default{ "image": "busybox" }

File transfer sidecar settings

Configure image, resources, and defaults for init/sidecar containers handling uploads/downloads.

defaultSpecobject

Default spec for file transfer containers

Overrides containerDefaultSpec for the init and sidecar containers that move files. Accepts Pod container fields such as securityContext, volumeMounts, resources, and env; useful for hardening or adding mounts used only by file transfer helpers.

Example: fileSidecar: defaultSpec: securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true volumeMounts: - name: tmp mountPath: /tmp

imagestring
Defaultbusybox

Image for file sidecar

Container image used by the init container (uploads input files) and the sidecar (downloads output files). Defaults to busybox.

The image must provide, on its PATH: a POSIX shell (sh), test/[, and sleep — required by the polling script that waits for the transfer to complete before the container exits. find and wc are also used, on a best-effort basis, to verify that uploaded files were fully transferred; if they're missing, verification is skipped rather than failing the task.

resourcesobject

Configure sidecar resource requests/limits

Optional Kubernetes resources block applied to the file transfer sidecar.

inheritClusterConfigbooleanstring
Defaultfalse

Inherit cluster auto-config

When true, the client config is seeded from the ambient auto-config (in-cluster service account, kube config, env) before applying config, so tuning like config.enableHttp2 works in-cluster without re-declaring credentials. Only applies when config is set; default false.

labelsobject

Add custom pod labels

Merged with Kestra default flow/execution/taskrun labels.

namespacestring
Defaultdefault

Namespace for created pod

Defaults to 'default'; template expressions are allowed.

nodeSelectorobject

Node selector for scheduling

Assigns the pod to nodes matching the selector (see Assign Pod Nodes).

pluginDefaultsRefstring

Reference (ref) of the pluginDefaults to apply to this task runner.

podSpecobject

Additional YAML spec for the pod

Optional PodSpec merged into the generated pod; Kestra still sets the main container, restartPolicy=Never, volumes, and sidecars. Supports template expressions including {{ workingDir }} resolving to /kestra/working-dir when file I/O is enabled. To define a spec, refer to the using pods guide from the official Kubernetes documentation.

Any container listed under containers whose name is not "main" is preserved as a user-defined sidecar alongside the Kestra main container. A container named "main" is merged into the Kestra-built main container as a set of defaults — fields such as ports, env, and imagePullPolicy from the user entry are applied first, then containerSpec and Kestra-injected values (image, commands, env vars like WORKING_DIR) take precedence on collision. For settings that must apply to all containers (e.g. volumeMounts, per-container securityContext), use containerDefaultSpec rather than repeating them per container.

resources

Set main container resources

CPU and memory requests/limits for the main task container.

limit
cpustring
memorystring
request
cpustring
memorystring
resumebooleanstring
Defaulttrue

Resume existing pod on restart

Default true; reconnects to an existing pod for the same taskrun when found.

serviceAccountNamestring

Choose service account name

Uses the namespace default when omitted; must carry required RBAC.

syncWorkingDirectorybooleanstring
Defaultfalse

Whether to synchronize working directory from remote runner back to local one after run.

versionstring

Plugin Version

Defines the version of the plugin to use.

The version must follow the Semantic Versioning (SemVer) specification:

  • A single-digit MAJOR version (e.g., 1).
  • A MAJOR.MINOR version (e.g., 1.1).
  • A MAJOR.MINOR.PATCH version, optionally with any qualifier (e.g., 1.1.2, 1.1.0-SNAPSHOT).
waitForLogsstring
DefaultPT30S

Extra wait for late logs

Default 30s; allows log streaming to finish after containers exit.

waitUntilCompletionstring
DefaultPT1H

Wait for pod completion

Pod wall-clock timeout when task timeout is unset; default 1h.

waitUntilRunningstring
DefaultPT10M

Wait for pod to start

Max time for scheduling, image pull, and container start; default 10m.

Enterprise-only task runner that launches the container image as a Cloud Batch Job; requires Batch Jobs Editor and Logs Viewer roles. Uses a GCS bucket for staging input/namespace files and collecting outputs (required when using those features), polls every 5s by default, and times out after 1h unless the task timeout overrides it. Jobs default to deleting on completion and resuming existing jobs when labels match; Cloud Logging captures stdout/stderr and the worker resumes a running job if the worker restarts. This task runner is container-based so the containerImage property must be set. You need to have roles 'Batch Job Editor' (roles/batch.jobsEditor) and 'Logs Viewer' (roles/logging.viewer) to be able to use it.

To access the task's working directory, use the {{workingDir}} Pebble expression or the WORKING_DIR environment variable. Input files and namespace files will be available in this directory.

To generate output files you can either use the outputFiles task's property and create a file with the same name in the task's working directory, or create any file in the output directory which can be accessed by the {{outputDir}} Pebble expression or the OUTPUT_DIR environment variables.

To use inputFiles, outputFiles or namespaceFiles properties, make sure to set the bucket property. The bucket serves as an intermediary storage layer for the task runner. Input and namespace files will be uploaded to the cloud storage bucket before the task run. Similarly, the task runner will store outputFiles in this bucket during the task run. In the end, the task runner will make those files available for download and preview from the UI by sending them to internal storage.

The task runner will generate a folder in the configured bucket for each task run. You can access that folder using the {{bucketPath}} Pebble expression or the BUCKET_PATH environment variable.

Warning, contrarily to other task runners, this task runner didn't run the task in the working directory but in the root directory. You must use the {{workingDir}} Pebble expression or the WORKING_DIR environment variable to access files.

Note that when the Kestra Worker running this task is terminated, the batch job will still runs until completion, then after restarting, the Worker will resume processing on the existing job unless resume is set to false.

Example

Execute a Shell command.

yaml
id: new_shell
namespace: company.team

variables:
  projectId: "myproject"
  region: "europe-west2"

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.Batch
      projectId: "{{vars.projectId}}"
      region: "{{ vars.region} }"
      serviceAccount: "{{ secret('GOOGLE_SA') }}"
    commands:
      - echo "Hello World"

Pass input files to the task, execute a Shell command, then retrieve output files.

yaml
id: new_shell_with_file
namespace: company.team

inputs:
  - id: file
    type: FILE

variables:
  projectId: "myProject"
  region: "europe-west2"
  bucket: "myBucket"

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    inputFiles:
      data.txt: "{{ inputs.file }}"
    outputFiles:
      - out.txt
    containerImage: centos
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.Batch
      projectId: "{{ vars.projectId }}"
      region: "{{ vars.region }}"
      bucket: "{{ vars.bucket }}"
      serviceAccount: "{{ secret('GOOGLE_SA') }}"
    commands:
      - cp {{workingDir}}/data.txt {{workingDir}}/out.txt

Run a Python script to fetch environment information on Google Cloud with Google Batch

yaml
id: gcp_batch_runner
namespace: company.team

tasks:
  - id: scrape_environment_info
    type: io.kestra.plugin.scripts.python.Commands
    containerImage: ghcr.io/kestra-io/pydata:latest
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.Batch
      projectId: "{{ secret('GCP_PROJECT_ID') }}"
      region: "europe-west9"
      bucket: "{{ secret('GCS_BUCKET')}}"
      serviceAccount: "{{ secret('GOOGLE_SA') }}"
    commands:
      - python {{ workingDir }}/main.py
    namespaceFiles:
      enabled: true
    outputFiles:
      - environment_info.json
    inputFiles:
      main.py: |
        import platform
        import socket
        import sys
        import json
        from kestra import Kestra

        print("Hello from GCP Batch and kestra!")

        def print_environment_info():
            print(f"Host's network name: {platform.node()}")
            print(f"Python version: {platform.python_version()}")
            print(f"Platform information (instance type): {platform.platform()}")
            print(f"OS/Arch: {sys.platform}/{platform.machine()}")

            env_info = {
                "host": platform.node(),
                "platform": platform.platform(),
                "OS": sys.platform,
                "python_version": platform.python_version(),
            }
            Kestra.outputs(env_info)

            filename = '{{ workingDir }}/environment_info.json'
            with open(filename, 'w') as json_file:
                json.dump(env_info, json_file, indent=4)

        if __name__ == '__main__':
          print_environment_info()
region*Requiredstring

GCP region

Region where the Batch job runs.

type*Requiredobject
Possible Values
io.kestra.plugin.ee.gcp.runner.Batchio.kestra.plugin.gcp.runner.Batch
bucketstring

Staging GCS bucket

Bucket used to upload input/namespace files and retrieve outputs; required when using file transfer or {{outputDir}}.

completionCheckIntervalstring
DefaultPT5S

Completion poll interval

How often to check job status; defaults to PT5S. Lower for short jobs, higher to reduce API calls. Setting this property to a lower value will reduce the number of API calls Kestra makes to the remote service — keep that in mind in case you see API rate limit errors.

computeResource

Per-task compute resources

ComputeResource defines the amount of resources required for each task. Make sure your tasks have enough compute resources to successfully run. If you also define the types of resources for a job to use with the InstancePolicyOrTemplate field, make sure both fields are compatible with each other. Override CPU, memory, and boot disk per task (defaults: cpu 2000 milliCPU, memory 2048 MiB). Values must stay compatible with the chosen machine type or instance policy.

bootDiskstring

Extra boot disk size

Additional boot disk size per task (e.g., 10GiB).

cpustring

CPU in milliCPU

Defines the amount of CPU resources per task in milliCPU units. For example, 1000 corresponds to 1 vCPU per task. If undefined, the default value is 2000. If you also define the VM's machine type using the machineType property in InstancePolicy or inside the instanceTemplate in InstancePolicyOrTemplate, make sure the CPU resources for both fields are compatible with each other and with how many tasks you want to allow to run on the same VM at the same time. For example, if you specify the n2-standard-2 machine type, which has 2 vCPUs, you can set the cpu to no more than 2000. Alternatively, you can run two tasks on the same VM if you set the cpu to 1000 or less.

memorystring

Memory in MiB

Per-task memory request in MiB; defaults to 2048. Must stay within the chosen machine type or instance template. If you also define the VM's machine type using the machineType in InstancePolicy or inside the instanceTemplate in InstancePolicyOrTemplate, make sure the memory resources for both fields are compatible with each other and with how many tasks you want to allow to run on the same VM at the same time. For example, if you specify the n2-standard-2 machine type, which has 8 GiB of memory, you can set the memory to no more than 8192.

deletebooleanstring
Defaulttrue

Delete job after completion

Defaults to true; set false to inspect or resume but stale jobs may be reused.

entryPointarray
SubTypestring

Container entrypoint

Override container entrypoint command.

impersonatedServiceAccountstring

The GCP service account to impersonate

Service account email to impersonate for API calls. For Cloud Run runner, this value applies to API calls used to create and run the job (--impersonate-service-account equivalent). It does not set the job execution identity (--service-account).

lifecyclePoliciesarray

Lifecycle policy for failed tasks

Optional policy executed when conditions match; defaults to exit on code 0 and retry on non-zero up to max_retry_count.

actionstring
Possible Values
ACTION_UNSPECIFIEDRETRY_TASKFAIL_TASKUNRECOGNIZED

Action on task failures

Batch action to take when lifecycle conditions are met.

actionCondition

Failure conditions

Conditions that trigger the lifecycle action (exit codes, etc.).

exitCodesarray
SubTypeinteger

Exit codes triggering the action

Lifecycle action fires when the task exits with any listed code.

machineTypestring
Defaulte2-medium

Compute machine type

VM type for the Batch instance; defaults to e2-medium. See https://cloud.google.com/compute/docs/machine-types

maxCreateJobRetryCountintegerstring
Default2

Maximum number of retries when creating the batch job fails

maxRetryCountinteger
Minimum>= 0
Maximum<= 10

Max task retries

Batch retry count on failure; default 0 disables retries.

networkInterfacesarray

Network interfaces

List of networks/subnets to attach to the Batch VM.

network*Requiredstring

Network identifier with the format projects/HOST_PROJECT_ID/global/networks/NETWORK

subnetworkstring

Subnetwork identifier in the format projects/HOST_PROJECT_ID/regions/REGION/subnetworks/SUBNET

pluginDefaultsRefstring

Reference (ref) of the pluginDefaults to apply to this task runner.

projectIdstring

The GCP project ID

reservationstring

Compute reservation

Reservation resource name to target reserved capacity.

resumebooleanstring
Defaulttrue

Resume existing job

If true (default), reattach to a matching job instead of creating a new one.

scopesarray
SubTypestring
Default["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used

serviceAccountstring

The GCP service account key

Service account JSON key used to authenticate API calls. For Cloud Run runner job execution identity, this value is used as a fallback for --service-account when runtimeServiceAccount is not provided.

syncWorkingDirectorybooleanstring
Defaultfalse

Whether to synchronize working directory from remote runner back to local one after run.

versionstring

Plugin Version

Defines the version of the plugin to use.

The version must follow the Semantic Versioning (SemVer) specification:

  • A single-digit MAJOR version (e.g., 1).
  • A MAJOR.MINOR version (e.g., 1.1).
  • A MAJOR.MINOR.PATCH version, optionally with any qualifier (e.g., 1.1.2, 1.1.0-SNAPSHOT).
waitForLogIntervalstring
DefaultPT5S

Post-completion log wait

Quiet period after the job ends: Kestra keeps polling for new log entries until none have arrived for this long, then finalizes logs and outputs; defaults to PT5S.

waitUntilCompletionstring
DefaultPT1H

Job timeout

Maximum wall-clock duration before Batch times out the job; defaults to PT1H. Task timeout takes precedence.

Enterprise-only task runner that launches the container image as a Cloud Run Job; requires Cloud Run Developer (roles/run.developer) and Logs Viewer (roles/logging.viewer) roles. Uses a GCS bucket to stage input/namespace files and collect outputs (required when using those features), polls every 5s by default, and times out after 1h unless the task timeout overrides it. The container starts in the root directory (use {{workingDir}} / WORKING_DIR); jobs default to deleting on completion and resuming existing executions when labels match. To generate output files, you can either use the outputFiles task's property and create a file with the same name in the task's working directory, or create any file in the output directory which can be accessed by the {{outputDir}} Pebble expression or the OUTPUT_DIR environment variables.

To use inputFiles, outputFiles, or namespaceFiles properties, make sure to set the bucket property. The bucket serves as an intermediary storage layer for the task runner. Input and namespace files will be uploaded to the cloud storage bucket before the task run. Similarly, the task runner will store outputFiles in this bucket during the task run. In the end, the task runner will make those files available for download and preview from the UI by sending them to internal storage.

The task runner will generate a folder in the configured bucket for each task run. You can access that folder using the {{ bucketPath }} Pebble expression or the BUCKET_PATH environment variable.

Warning: contrary to other task runners, this task runner didn't run the task in the working directory but in the root directory. You must use the {{ workingDir }} Pebble expression or the WORKING_DIR environment variable to access files.

The waitUntilCompletion property maps directly to the GCP "Task timeout" field (visible in the GCP console under Task capacity). It is applied both to the job template and to the per-run override, so the Cloud Run task will be forcibly terminated by GCP when this duration elapses. The Kestra task-level timeout property takes precedence over waitUntilCompletion when set.

Note that when the Kestra Worker running this task is terminated, the Cloud Run Job will still run until completion.

Example

Execute a Shell command.

yaml
id: new-shell
namespace: company.team

variables:
  projectId: "projectId"
  region: "europe-west2"

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.CloudRun
      projectId: "{{ vars.projectId }}"
      region: "{{ vars.region }}"
      serviceAccount: "{{ secret('GOOGLE_SA') }}"
    commands:
      - echo "Hello World"

Pass input files to the task, execute a Shell command, then retrieve output files.

yaml
id: new-shell-with-file
namespace: company.team

inputs:
  - id: file
    type: FILE

variables:
  projectId: "projectId"
  bucket: "bucket"
  region: "europe-west2"

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    inputFiles:
      data.txt: "{{ inputs.file }}"
    outputFiles:
      - out.txt
    containerImage: centos
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.CloudRun
      projectId: "{{ vars.projectId }}"
      region: "{{ vars.region }}"
      bucket: "{{ vars.bucket }}"
      serviceAccount: "{{ secret('GOOGLE_SA') }}"
    commands:
      - cp {{ workingDir }}/data.txt {{ workingDir }}/out.txt

Execute a task with Direct VPC Egress.

yaml
id: shell-direct-vpc-egress
namespace: company.team

variables:
  projectId: "projectId"
  region: "europe-west1"
  network: "projects/projectId/global/networks/my-vpc"
  subnetwork: "projects/projectId/regions/europe-west1/subnetworks/my-subnet"

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.CloudRun
      projectId: "{{ vars.projectId }}"
      region: "{{ vars.region }}"
      network: "{{ vars.network }}"
      subnetwork: "{{ vars.subnetwork }}"
      vpcEgress: ALL_TRAFFIC
      serviceAccount: "{{ secret('GOOGLE_SA') }}"
    commands:
      - echo "Hello from Direct VPC Egress"

Mount GCS buckets as volumes (read-only or read-write).

yaml
id: shell-with-gcs-volume
namespace: company.team

variables:
  projectId: "projectId"
  region: "europe-west1"

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.CloudRun
      projectId: "{{ vars.projectId }}"
      region: "{{ vars.region }}"
      serviceAccount: "{{ secret('GOOGLE_SA') }}"
      volumes:
        - bucket: my-reference-data-bucket
          mountPath: /data
          readOnly: true
        - bucket: my-output-bucket
          mountPath: /output
    commands:
      - ls /data
      - ls /output
region*Requiredstring

GCP region

Region where the Cloud Run job executes.

type*Requiredobject
Possible Values
io.kestra.plugin.ee.gcp.runner.CloudRunio.kestra.plugin.gcp.runner.CloudRun
bucketstring

Staging GCS bucket

Bucket used to upload input/namespace files and retrieve outputs; required when using file transfer or {{outputDir}}.

completionCheckIntervalstring
DefaultPT5S

Completion poll interval

How often to check execution status; defaults to PT5S. Lower for short jobs, higher to reduce API calls.

deletebooleanstring
Defaulttrue

Delete job after completion

Defaults to true; set false to inspect runs but stale jobs may be reused.

impersonatedServiceAccountstring

The GCP service account to impersonate

Service account email to impersonate for API calls. For Cloud Run runner, this value applies to API calls used to create and run the job (--impersonate-service-account equivalent). It does not set the job execution identity (--service-account).

maxRetriesintegerstring
Default3

Max Cloud Run retries

Number of execution retries; defaults to 3.

networkstring

VPC network

VPC network for Direct VPC Egress, for example projects/my-project/global/networks/my-vpc; cannot be used with vpcAccessConnector.

pluginDefaultsRefstring

Reference (ref) of the pluginDefaults to apply to this task runner.

projectIdstring

The GCP project ID

resources

Container resource limits

CPU and memory limits for the Cloud Run container. If unset, Cloud Run defaults apply.

cpustring

CPU limit

CPU limit for the Cloud Run container. Examples: 1, 2, 4, 8 (vCPUs) or 1000m (millicores). See https://cloud.google.com/run/docs/configuring/cpu

memorystring

Memory limit

Memory limit for the Cloud Run container. Examples: 512Mi, 1Gi, 2Gi, 4Gi. See https://cloud.google.com/run/docs/configuring/memory-limits

resumebooleanstring
Defaulttrue

Resume existing job

If true (default), reattach to a matching job instead of creating a new one.

runtimeServiceAccountstring

Cloud Run runtime service account

Service account email used as the Cloud Run Job execution identity (--service-account equivalent). This only controls runtime identity for the job's container and does not affect API authentication. If unset, the runner falls back to serviceAccount for backward compatibility.

scopesarray
SubTypestring
Default["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used

serviceAccountstring

The GCP service account key

Service account JSON key used to authenticate API calls. For Cloud Run runner job execution identity, this value is used as a fallback for --service-account when runtimeServiceAccount is not provided.

subnetworkstring

VPC subnetwork

VPC subnetwork for Direct VPC Egress, for example projects/my-project/regions/europe-west1/subnetworks/my-subnet; cannot be used with vpcAccessConnector.

syncWorkingDirectorybooleanstring

Whether to synchronize working directory from remote runner back to local one after run.

versionstring

Plugin Version

Defines the version of the plugin to use.

The version must follow the Semantic Versioning (SemVer) specification:

  • A single-digit MAJOR version (e.g., 1).
  • A MAJOR.MINOR version (e.g., 1.1).
  • A MAJOR.MINOR.PATCH version, optionally with any qualifier (e.g., 1.1.2, 1.1.0-SNAPSHOT).
volumesarray

Additional GCS bucket volumes

List of GCS buckets to mount as volumes inside the container. Each entry requires a bucket name and a container mount path. Set readOnly: true for read-only access; omit or set readOnly: false (the default) for read-write access.

bucket*Requiredstring

GCS bucket name

Name of the Cloud Storage bucket to mount.

mountPath*Requiredstring

Container mount path

Absolute path inside the container where the bucket will be mounted.

readOnlybooleanstring
Defaultfalse

Read-only

If true, the volume is mounted read-only. Defaults to false.

vpcAccessConnectorstring

VPC Access Connector

Full resource name for egress connector, e.g. projects/my-project/locations/europe-west1/connectors/my-connector.

vpcEgressstring
Possible Values
VPC_EGRESS_UNSPECIFIEDALL_TRAFFICPRIVATE_RANGES_ONLYUNRECOGNIZED

VPC egress setting

PRIVATE_RANGES_ONLY or ALL_TRAFFIC; requires either vpcAccessConnector, network, or subnetwork.

waitForLogIntervalstring
DefaultPT5S

Post-completion log wait

Quiet period after the job ends: Kestra keeps polling for new log entries until none have arrived for this long, then finalizes logs and outputs; defaults to PT5S.

waitUntilCompletionstring
DefaultPT1H

Task timeout

Maps to the GCP "Task timeout" field visible in the GCP console under Task capacity. Controls both the GCP-enforced task timeout (applied to the job template and the per-run override) and the Kestra polling timeout — the Cloud Run task is forcibly terminated by GCP when this duration elapses. The Kestra task-level timeout property takes precedence over this value when set. Defaults to PT1H. GCP maximum is 168 hours (PT168H).

Enterprise-only task runner that runs task scripts directly on a Compute Engine VM instance — no SSH, no IAP tunnel. The script is injected as the instance's startup-script metadata, its stdout/stderr is mirrored by the guest environment to the serial console (port 1) and streamed back from there, and completion is detected through a guest attribute (kestra/status) the script writes when it exits. This requires enable-guest-attributes (set automatically by this runner) and an instance image that provides bash, curl, and python3 — true of the default Debian/Ubuntu Compute Engine images.

You need the roles/compute.instanceAdmin.v1 role (or equivalent fine-grained permissions) to create, start, stop, delete instances, read guest attributes, and read the serial console, plus roles/iam.serviceAccountUser if the instance runs under a dedicated service account.

By default a new instance is created from instanceConfig — a raw JSON object matching the instances.insert request body (at minimum disks and networkInterfaces) — then stopped and/or deleted after the run according to stopInstance/deleteInstance. Set instanceName instead to target an already existing instance: since a metadata startup-script only runs on boot, ComputeEngine reboots (instances.reset) that instance to (re)trigger the script if it is already running — this interrupts anything else the instance was doing, so never point two concurrent Kestra executions at the same instanceName. ComputeEngine never deletes an instance it did not create, regardless of deleteInstance; use stopInstance if you also want it stopped.

Uses a GCS bucket for staging inputFiles/namespaceFiles and collecting outputFiles (required when using those features). The instance's own network must be able to reach Cloud Storage and the metadata server (an external IP, Cloud NAT, or Private Google Access all work) and its service account needs at least the https://www.googleapis.com/auth/devstorage.read_write scope for that bucket.

To access the task's working directory, use the {{workingDir}} Pebble expression or the WORKING_DIR environment variable — unlike the Batch and Cloud Run runners, commands here already execute from that directory. Output files can be created with the same name as listed in outputFiles anywhere in that directory, or under {{outputDir}} (OUTPUT_DIR) for automatic capture.

If the Kestra Worker running this task is terminated, the instance keeps running until completion; when resume is true (default) the Worker reattaches to it after restarting instead of creating a duplicate.

Example

Create a Compute Engine VM, run a Shell command, then delete the VM.

yaml
id: compute_engine_shell
namespace: company.team

variables:
  projectId: "my-project"
  zone: "europe-west1-b"

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.ComputeEngine
      projectId: "{{ vars.projectId }}"
      zone: "{{ vars.zone }}"
      serviceAccount: "{{ secret('GOOGLE_SA') }}"
      instanceConfig:
        disks:
          - boot: true
            initializeParams:
              sourceImage: "projects/debian-cloud/global/images/family/debian-12"
        networkInterfaces:
          - network: "projects/{{ vars.projectId }}/global/networks/default"
            accessConfigs:
              - type: "ONE_TO_ONE_NAT"
    commands:
      - echo "Hello World"

Pass input files to an existing named instance, execute a Shell command, then retrieve output files.

yaml
id: compute_engine_existing_instance
namespace: company.team

inputs:
  - id: file
    type: FILE

variables:
  projectId: "my-project"
  zone: "europe-west1-b"
  bucket: "my-bucket"

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    inputFiles:
      data.txt: "{{ inputs.file }}"
    outputFiles:
      - out.txt
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.ComputeEngine
      projectId: "{{ vars.projectId }}"
      zone: "{{ vars.zone }}"
      bucket: "{{ vars.bucket }}"
      instanceName: "my-persistent-vm"
      stopInstance: false
      deleteInstance: false
      serviceAccount: "{{ secret('GOOGLE_SA') }}"
    commands:
      - cp {{ workingDir }}/data.txt {{ workingDir }}/out.txt
type*Requiredobject
Possible Values
io.kestra.plugin.ee.gcp.runner.ComputeEngineio.kestra.plugin.gcp.runner.ComputeEngine
zone*Requiredstring

GCP zone

Zone the Compute Engine instance runs in, for example europe-west1-b.

bucketstring

Staging GCS bucket

Bucket used to upload input/namespace files and retrieve outputs; required when using file transfer or {{outputDir}}.

completionCheckIntervalstring
DefaultPT5S

Completion poll interval

How often to check the completion guest attribute and the serial console for new log lines; defaults to PT5S. Setting this property to a lower value will reduce the number of API calls Kestra makes to the remote service — keep that in mind in case you see API rate limit errors.

deleteInstancebooleanstring
Defaulttrue

Delete the instance after completion

Defaults to true; set false to inspect or resume but stale instances may be reused. Never deletes an instance targeted through instanceName — ComputeEngine never deletes an instance it did not create — but it still controls whether that run's staged GCS blob prefix is cleaned up, regardless of instance ownership.

impersonatedServiceAccountstring

The GCP service account to impersonate

Service account email to impersonate for API calls. For Cloud Run runner, this value applies to API calls used to create and run the job (--impersonate-service-account equivalent). It does not set the job execution identity (--service-account).

instanceConfigobject

New instance configuration

Raw JSON object used to create a new instance, in the same shape as the instances.insert request body — at minimum disks (with initializeParams.sourceImage) and networkInterfaces. Required unless instanceName is set. Any name field is ignored: ComputeEngine always generates the instance name from the flow/task/execution. The instance's network must be able to reach Cloud Storage and the metadata server (an external IP, Cloud NAT, or Private Google Access all work).

instanceNamestring

Existing instance name

Name of an already existing instance to run the task on, instead of creating a new one. Since a metadata startup-script only runs on boot, ComputeEngine reboots (instances.reset) the instance to (re)trigger it if it is already running when the task starts — this is destructive to anything else the instance is doing. Never target the same instanceName from two concurrent Kestra executions: the second run's reboot kills the first run's in-flight script. When set, instanceConfig is ignored. deleteInstance never deletes an instance targeted through instanceName — ComputeEngine never deletes an instance it did not create — but it still controls whether that run's staged GCS blob prefix is cleaned up, regardless of instance ownership.

machineTypestring
Defaulte2-medium

Compute machine type

VM type for a newly created instance; defaults to e2-medium. Always overrides any machineType set inside instanceConfig. Only applies when creating a new instance. See https://cloud.google.com/compute/docs/machine-types

pluginDefaultsRefstring

Reference (ref) of the pluginDefaults to apply to this task runner.

projectIdstring

The GCP project ID

resumebooleanstring
Defaulttrue

Resume an existing instance

If true (default) and instanceName is not set, reattach to a matching Kestra-created instance instead of creating a new one — used when the Worker restarts mid-run.

scopesarray
SubTypestring
Default["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used

serviceAccountstring

The GCP service account key

Service account JSON key used to authenticate API calls. For Cloud Run runner job execution identity, this value is used as a fallback for --service-account when runtimeServiceAccount is not provided.

stopInstancebooleanstring
Defaulttrue

Stop the instance after completion

Defaults to true. Ignored if deleteInstance is also true (deleting supersedes stopping), and never applies to an instance targeted through instanceName.

syncWorkingDirectorybooleanstring
Defaultfalse

Whether to synchronize working directory from remote runner back to local one after run.

versionstring

Plugin Version

Defines the version of the plugin to use.

The version must follow the Semantic Versioning (SemVer) specification:

  • A single-digit MAJOR version (e.g., 1).
  • A MAJOR.MINOR version (e.g., 1.1).
  • A MAJOR.MINOR.PATCH version, optionally with any qualifier (e.g., 1.1.2, 1.1.0-SNAPSHOT).
waitForLogIntervalstring
DefaultPT5S

Post-completion log wait

Quiet period after the task ends: Kestra keeps polling the serial console for new log lines until none have arrived for this long, then finalizes logs and outputs; defaults to PT5S.

waitUntilCompletionstring
DefaultPT1H

Completion timeout

Maximum wall-clock duration to wait for the task script to complete before timing out; defaults to PT1H. Task timeout takes precedence.

Defaulttrue

Whether to automatically download and install 'uv' when it is missing from the worker

When enabled (default), if 'uv' cannot be found on the worker, it is downloaded from a pinned, checksum-verified installer and installed automatically. Disable this on locked-down or air-gapped workers where remote downloads are not allowed; in that case, 'uv' must be pre-installed on the worker (or exposed via the 'UV_PATH' environment variable). When disabled and 'uv' is absent, the task fails fast with an actionable error instead of silently falling back to PIP.

The expected SHA-256 checksum of the pinned 'uv' installer script

Used to verify the integrity of the installer downloaded from https://astral.sh/uv/<uvInstallerVersion>/install.sh before executing it. Defaults to the checksum matching the bundled 'uvInstallerVersion'. Override this together with 'uvInstallerVersion' when bumping the pinned 'uv' version. The download is rejected and the task fails if the checksums do not match — it never falls back to executing an unverified installer or degrades to PIP.

The pinned version of 'uv' to download when it is missing from the worker

Only used when 'uvAutoInstallEnabled' is true and 'uv' cannot be found on the worker. Defaults to a version bundled with this plugin. Override this to bump the auto-installed 'uv' version without waiting for a plugin release; when doing so, also set 'uvInstallerSha256' to the matching checksum.

Default0

The exit code of the entire flow execution.

SubTypestring

The output files' URIs in Kestra's internal storage.

Definitions

The values extracted from executed commands using the Kestra outputs format.

Number of records converted

Number of records converted