AWS Batch

AWS Batch

Certified
Enterprise Edition

Task runner that executes a task inside a job in AWS Batch

This plugin is only available in the Enterprise Edition (EE).

This task runner supports ECS (Fargate or EC2) and EKS compute environments.

Make sure to set the containerImage property because this runner runs the task in a container.

To access the task's working directory, use the {{ workingDir }} Pebble expression or the WORKING_DIR environment variable. This directory will contain all input files and namespace files (if enabled).

To generate output files you can either use the outputFiles task property and create a file with the same name in the task's working directory, or create any file in the output directory which can be accessed using the {{ outputDir }} Pebble expression or the OUTPUT_DIR environment variable.

To use inputFiles, outputFiles or namespaceFiles properties, make sure to set the bucket property. The bucket serves as an intermediary storage layer for the task runner. Input and namespace files will be uploaded to the cloud storage bucket before the task run starts. Similarly, the task runner will store outputFiles in this bucket during the task run. In the end, the task runner will make those files available for download and preview from the UI by sending them to internal storage.

The task runner will generate a folder in the configured bucket for each task run. You can access that folder using the {{bucketPath}} Pebble expression or the BUCKET_PATH environment variable.

Note that this task runner executes the task in the root directory. You need to use the {{ workingDir }} Pebble expression or the WORKING_DIR environment variable to access files in the task's working directory.

Note that when the Kestra Worker running this task is terminated, the batch job will still runs until completion, then after restarting, the Worker will resume processing on the existing job unless resume is set to false.

This task runner will return with an exit code according to the following mapping:

  • SUCCEEDED: 0
  • FAILED: 1
  • RUNNING: 2
  • RUNNABLE: 3
  • PENDING: 4
  • STARTING: 5
  • SUBMITTED: 6
  • OTHER: -1

To avoid zombie containers in ECS, you can set the timeout property on the task, and Kestra will terminate the batch job if the task is not completed within the specified duration.

By default, this task runner streams job logs in near real-time via CloudWatch Logs Live Tail, which is priced per GB of log data scanned/streamed and requires long-lived AWS credentials for the duration of the job. For cost-sensitive workloads or when using short-lived/role-chained credentials (which can expire mid-job and interrupt the log stream), set streamLogs to false to disable Live Tail entirely — job polling and exit-code mapping are unaffected either way.

**EKS-specific notes: **

  • CloudWatch logs: Log streaming requires Fluent Bit (or the CloudWatch agent) installed on the EKS cluster and configured to forward pod logs to the /aws/batch/job log group with the job name as the log stream prefix. Without this, no logs will appear in Kestra. This is a cluster-level concern managed by the operator, not by Kestra. A runtime warning is emitted when an EKS compute environment is detected.
  • /bin/sh requirement: When outputFiles, {{ outputDir }}, or syncWorkingDirectory are used, the main container command is wrapped in /bin/sh -c. The container image must include /bin/sh. Most standard images (Ubuntu, Amazon Linux, Alpine) do; distroless or scratch images do not. A runtime warning is emitted as a reminder.
  • Use serviceAccountName with IRSA for IAM authorization instead of taskRoleArn.
  • executionRoleArn and taskRoleArn are ignored for EKS compute environments.
yaml
type: io.kestra.plugin.ee.aws.runner.Batch

Execute a Shell command in a container on ECS Fargate.

yaml
id: run_container
namespace: company.team

variables:
  region: eu-west-2
  computeEnvironmentArn: "arn:aws:batch:eu-west-2:123456789012:compute-environment/kestraFargateEnvironment"

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.aws.runner.Batch
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
      region: "{{ vars.region }}"
      computeEnvironmentArn: "{{ vars.computeEnvironmentArn }}"
    commands:
      - echo "Hello World"

Pass input files to the task, execute a Shell command, then retrieve the output files.

yaml
id: container_with_input_files
namespace: company.team

inputs:
  - id: file
    type: FILE

variables:
  region: eu-west-2
  computeEnvironmentArn: "arn:aws:batch:eu-central-1:123456789012:compute-environment/kestraFargateEnvironment"

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    inputFiles:
      data.txt: "{{ inputs.file }}"
    outputFiles:
      - out.txt
    containerImage: centos
    taskRunner:
      type: io.kestra.plugin.ee.aws.runner.Batch
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
      region: "{{ vars.region }}"
      bucket: "{{ vars.bucket }}"
      computeEnvironmentArn: "{{ vars.computeEnvironmentArn }}"
    commands:
      - cp {{ workingDir }}/data.txt {{ workingDir }}/out.txt

Run a Python script to fetch environment information on AWS ECS Fargate with AWS Batch

yaml
id: aws_batch_runner
namespace: company.team

variables:
  compute_environment_arn: arn:aws:batch:us-east-1:123456789:compute-environment/kestra
  job_queue_arn: arn:aws:batch:us-east-1:123456789:job-queue/kestra
  execution_role_arn: arn:aws:iam::123456789:role/ecsTaskExecutionRole
  task_role_arn: arn:aws:iam::123456789:role/ecsTaskRole

tasks:
  - id: send_data
    type: io.kestra.plugin.scripts.python.Script
    containerImage: ghcr.io/kestra-io/pydata:latest
    taskRunner:
      type: io.kestra.plugin.ee.aws.runner.Batch
      region: us-east-1
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
      computeEnvironmentArn: "{{ vars.compute_environment_arn }}"
      jobQueueArn: "{{ vars.job_queue_arn }}"
      executionRoleArn: "{{ vars.execution_role_arn }}"
      taskRoleArn: "{{ vars.task_role_arn }}"
      bucket: kestra-us
    script: |
      import platform
      import socket
      import sys

      print("Hello from AWS Batch and kestra!")

      def print_environment_info():
          print(f"Host's network name: {platform.node()}")
          print(f"Python version: {platform.python_version()}")
          print(f"Platform information (instance type): {platform.platform()}")
          print(f"OS/Arch: {sys.platform}/{platform.machine()}")

          try:
              hostname = socket.gethostname()
              ip_address = socket.gethostbyname(hostname)
              print(f"Host IP Address: {ip_address}")
          except socket.error as e:
              print("Unable to obtain IP address.")

      if __name__ == '__main__':
          print_environment_info()

Execute a Shell command on AWS Batch with an EKS compute environment.

yaml
id: run_container_on_eks
namespace: company.team

variables:
  region: us-east-1
  compute_environment_arn: arn:aws:batch:us-east-1:123456789:compute-environment/kestraEksEnvironment
  job_queue_arn: arn:aws:batch:us-east-1:123456789:job-queue/kestraEksQueue

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    containerImage: amazonlinux:2
    taskRunner:
      type: io.kestra.plugin.ee.aws.runner.Batch
      region: "{{ vars.region }}"
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
      computeEnvironmentArn: "{{ vars.compute_environment_arn }}"
      jobQueueArn: "{{ vars.job_queue_arn }}"
      serviceAccountName: kestra-sa
    commands:
      - echo "Hello from AWS Batch on EKS"
Properties

Compute environment in which to run the job

AWS region with which the SDK should communicate

Access Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

S3 Bucket to upload (inputFiles and namespaceFiles) and download (outputFiles) files

It's mandatory to provide a bucket if you want to use such properties.

DefaultPT5S

Determines how often Kestra should poll the container for completion. By default, the task runner checks every 5 seconds whether the job is completed. You can set this to a lower value (e.g. PT0.1S = every 100 milliseconds) for quick jobs and to a lower threshold (e.g. PT1M = every minute) for long-running jobs. Setting this property to a lower value will reduce the number of API calls Kestra makes to the remote service — keep that in mind in case you see API rate limit errors

Defaulttrue

Whether the job should be deleted upon completion

Warning, if the job is not deleted, a retry of the task could resume an old failed attempt of the job.

The endpoint with which the SDK should communicate

This property allows you to use a different S3 compatible storage backend.

Execution role for the AWS Batch job

Mandatory if the compute environment is ECS Fargate. See the AWS documentation for more details. Ignored for EKS compute environments.

Job queue to use to submit jobs (ARN). If not specified, the task runner will create a job queue — keep in mind that this can lead to a longer execution

Reference (ref) of the pluginDefaults to apply to this task runner.

Default{ "request": { "memory": "2048", "cpu": "1" } }

Custom resources for the container

For ECS Fargate, see the AWS documentation for supported CPU/memory combinations. For EKS, values follow Kubernetes resource format: CPU as a decimal (e.g. 0.5, 1) and memory as an integer in MiB (e.g. 2048).

Definitions
request*Required
cpu*Requiredstring
memory*Requiredstring
Defaulttrue

Whether to reconnect to the current job if it already exists

Secret Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

Kubernetes service account name for the EKS pod

Used with IRSA (IAM Roles for Service Accounts) for IAM authorization in EKS compute environments. Ignored for ECS.

AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

Resources for sidecar containers

Resources used by the internal inputFiles and outputFiles containers (S3 upload/download). If not set, defaults to minimal valid values depending on the compute environment (Fargate: 0.25 vCPU / 512 MiB; EC2: 1 vCPU / 128 MiB). Note: on Fargate, AWS Batch validates resources at the task level; sidecar resources are therefore subtracted from the main container to keep the overall task resources equal to resources.request.

Definitions
request*Required
cpu*Requiredstring
memory*Requiredstring
Defaulttrue

Whether to stream job logs in near real-time using CloudWatch Logs Live Tail

When true (default), the task runner starts a CloudWatch Logs Live Tail session so job logs appear in the Kestra execution as they are produced. Live Tail is priced per GB of log data scanned/streamed and requires the AWS credentials to remain valid for the entire job duration; if short-lived or role-chained credentials expire mid-job, the log stream is interrupted and the task fails even if the underlying AWS Batch job succeeds. Set this to false to skip Live Tail entirely — no CloudWatch API calls are made — job polling and exit-code mapping are unaffected. Job container logs are still written to the /aws/batch/job CloudWatch log group by the awslogs log driver (ECS) or Fluent Bit (EKS) regardless of this setting; only near-real-time streaming into Kestra is disabled. Note: unlike the unrelated streamLogs property on io.kestra.plugin.ee.aws.batch.Run (which defaults to false), this defaults to true because Live Tail is this task runner's only way to surface job output in the Kestra execution as it happens.

The AWS STS endpoint with which the SDKClient should communicate

AWS STS Role

The Amazon Resource Name (ARN) of the role to assume. If set the task will use the StsAssumeRoleCredentialsProvider. If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

AWS STS External Id

A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.

DefaultPT15M

AWS STS Session duration

The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.

AWS STS Session name

This property is only used when an stsRoleArn is defined.

Defaultfalse

Whether to synchronize working directory from remote runner back to local one after run.

Task role to use within the container

Needed if you want to authenticate with AWS CLI within your container. Ignored for EKS compute environments.

Plugin Version

Defines the version of the plugin to use.

The version must follow the Semantic Versioning (SemVer) specification:

  • A single-digit MAJOR version (e.g., 1).
  • A MAJOR.MINOR version (e.g., 1.1).
  • A MAJOR.MINOR.PATCH version, optionally with any qualifier (e.g., 1.1.2, 1.1.0-SNAPSHOT).
DefaultPT1H

The maximum duration to wait for the job completion unless the task timeout property is set which will take precedence over this property

AWS Batch will automatically timeout the job upon reaching that duration and the task will be marked as failed.

Default20480

Maximum storage in MiB for the task's working directory on EKS compute environments

Size limit of the working-directory volume backing the EKS pod, in mebibytes (MiB). Raise it for tasks that sync large input or output files. Ignored for ECS compute environments.

More Plugins in this Category

Core Plugins and tasks
Core Plugins and tasks

Tasks that provide Kestra's built-in orchestration, I/O, and observability capabilities.

Core

122 tasks

398 blueprints

NetBox (EE)
NetBox
Enterprise

Enterprise NetBox tasks to create, update, and query devices, IP addresses, and sites over the NetBox API.

Infrastructure

7 tasks

4 blueprints