Execute Kestra Tasks as AWS Batch Jobs on ECS Fargate or EC2

Run tasks as AWS ECS Fargate or EC2 containers using AWS Batch.

Offload tasks to AWS Batch

To launch tasks on AWS Batch, you need to understand three key concepts:

1. Compute environment — mandatory; it won’t be created by the task. The compute environment defines the infrastructure for your tasks and can be either ECS Fargate or EC2.
2. Job queue — optional; it will be created by the task if not specified. Creating a queue adds some latency to the script’s runtime.
3. Job — created by the task runner; contains information about the image, commands, and resources to use. In AWS ECS terminology, it’s the task definition.

How does the AWS Batch task runner work?

In order to support inputFiles, namespaceFiles, and outputFiles, the AWS Batch task runner currently relies on multi-container ECS jobs and creates three containers for each job:

1. A before-container that uploads input files to S3.
2. The main container that fetches input files into the {{ workingDir }} directory and runs the task.
3. An after-container that fetches output files using outputFiles to make them available from the Kestra UI for download and preview.

Since the working directory of the container isn’t known in advance, you must define the working and output directories explicitly. For example, use cat {{ workingDir }}/myFile.txt instead of cat myFile.txt.

Exit codes

The task runner maps AWS Batch job statuses to exit codes as follows:

Minimum permissions required

To submit and monitor AWS Batch jobs, the IAM principal used by Kestra needs permission to create, tag, inspect, and clean up Batch job definitions and jobs. It also needs permission to pass the ECS roles used by the job and to read the AWS Batch log group.

The following policy is the minimum set required by the task runner:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "logs:DescribeLogGroups",
        "batch:TagResource",
        "batch:SubmitJob",
        "batch:RegisterJobDefinition",
        "batch:ListJobs",
        "batch:DescribeJobs",
        "batch:DescribeJobDefinitions",
        "batch:DescribeComputeEnvironments",
        "batch:DeregisterJobDefinition",
        "batch:TerminateJob",
        "batch:CreateJobQueue",
        "batch:UpdateJobQueue",
        "batch:DeleteJobQueue",
        "batch:DescribeJobQueues"
      ],
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Action": [
        "iam:PassRole"
      ],
      "Effect": "Allow",
      "Resource": [
        "<executionRoleArn>",
        "<serviceRoleArn>",
        "<taskRoleArn>"
      ]
    },
    {
      "Action": [
        "logs:StartLiveTail"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:logs:eu-central-1:<accountId>:log-group:/aws/batch/job"
    }
  ]
}

Replace <executionRoleArn>, <serviceRoleArn>, <taskRoleArn>, and <accountId> with the values from your AWS account. If you use a different region, update the CloudWatch Logs ARN accordingly.

S3 permissions when using bucket

When you set the bucket property, the Kestra worker itself (not the ECS task container) uploads inputFiles and namespaceFiles to S3 before the job starts and downloads outputFiles after it finishes. It also deletes the working-directory prefix from the bucket on cleanup. The Kestra IAM principal therefore needs the following additional permissions when bucket is configured:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

The ECS task container separately needs S3 access via its taskRoleArn to read input files and write output files at runtime. Refer to the Create the ecsTaskRole IAM role section for the task-level policy.

Resource sizing

Default resources

By default, each job runs with 1 vCPU and 2048 MiB of memory. Override this with the resources property:

taskRunner:
  type: io.kestra.plugin.ee.aws.runner.Batch
  # ...
  resources:
    request:
      cpu: "2"
      memory: "4096"

Fargate CPU and memory constraints

AWS Fargate enforces strict combinations of vCPU and memory. The task runner validates these at runtime and will throw an error if an invalid combination is used.

For EC2 compute environments, the vCPU value must be a whole integer (e.g. "1", "2") and must be ≥ 1.

Sidecar container resources

When inputFiles, namespaceFiles, or outputFiles are used, the task runner adds sidecar containers that handle S3 file transfers. Default sidecar resources are:

• ECS Fargate: 0.25 vCPU / 512 MiB
• ECS EC2: 1 vCPU / 128 MiB

On Fargate, AWS Batch enforces resource limits at the task level. To keep the overall task resources equal to the value set in resources.request, the sidecar resources are automatically subtracted from the main container. For example, with resources.request = 1 vCPU / 2048 MiB and one sidecar at the default 0.25 vCPU / 512 MiB, the main container will receive 0.75 vCPU / 1536 MiB.

If your resources.request is too small to accommodate the sidecars, the task runner will throw an error at startup. You can either increase resources.request or override sidecar sizing with sidecarResources:

taskRunner:
  type: io.kestra.plugin.ee.aws.runner.Batch
  # ...
  resources:
    request:
      cpu: "1"
      memory: "2048"
  sidecarResources:
    request:
      cpu: "0.25"
      memory: "512"

How to run tasks on AWS ECS Fargate

The example below demonstrates how to use the AWS Batch task runner to offload Python scripts to a serverless container running on AWS ECS Fargate:

id: aws_batch_runner
namespace: company.team

tasks:
  - id: scrape_environment_info
    type: io.kestra.plugin.scripts.python.Script
    containerImage: ghcr.io/kestra-io/pydata:latest
    taskRunner:
      type: io.kestra.plugin.ee.aws.runner.Batch
      region: eu-central-1
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
      computeEnvironmentArn: "arn:aws:batch:eu-central-1:707969873520:compute-environment/kestraFargateEnvironment"
      jobQueueArn: "arn:aws:batch:eu-central-1:707969873520:job-queue/kestraJobQueue"
      executionRoleArn: "arn:aws:iam::707969873520:role/kestraEcsTaskExecutionRole"
      taskRoleArn: arn:aws:iam::707969873520:role/ecsTaskRole
      bucket: kestra-product-de
    namespaceFiles:
      enabled: true
    outputFiles:
      - "*.json"
    script: |
      import platform
      import socket
      import sys
      import json
      from kestra import Kestra

      print("Hello from AWS Batch and kestra!")


      def print_environment_info():
          print(f"Host's network name: {platform.node()}")
          print(f"Python version: {platform.python_version()}")
          print(f"Platform information (instance type): {platform.platform()}")
          print(f"OS/Arch: {sys.platform}/{platform.machine()}")

          env_info = {
              "host": platform.node(),
              "platform": platform.platform(),
              "OS": sys.platform,
              "python_version": platform.python_version(),
          }
          Kestra.outputs(env_info)

          filename = "{{ workingDir }}/environment_info.json"
          with open(filename, "w") as json_file:
              json.dump(env_info, json_file, indent=4)


      if __name__ == "__main__":
          print_environment_info()

Full step-by-step guide: setting up AWS Batch from scratch

To use the AWS Batch task runner, you must configure resources in your AWS account. You can set up the environment in two ways:

1. Using Terraform to provision all necessary resources using a simple terraform apply command.
2. Creating the resources step by step from the AWS Management Console.

Before you begin

You will need:

1. An AWS account.
2. A Kestra Enterprise Edition instance running version 0.18.0 or later with AWS credentials stored as secrets.

Terraform setup

Follow the instructions in the aws-batch README in the terraform-deployments-templates repository to provision resources using Terraform. You can also use this blueprint, which creates all required resources in a single Kestra workflow execution.

Here is a list of resources that will be created:

• AWS Security Group: a security group for AWS Batch jobs with egress to the internet (required to be able to download public Docker images in your script tasks).
• AWS IAM Roles and Policies: IAM roles and policies for AWS Batch and ECS Task Execution, including permissions for S3 access (S3 is used to store input and output files for container access).
• AWS Batch Compute Environment: a managed ECS Fargate compute environment named kestraFargateEnvironment.
• AWS Batch Job Queue: a job queue named kestraJobQueue for submitting batch jobs.

AWS Management Console setup

Create the ecsTaskExecutionRole IAM role

Create an execution role that allows AWS Batch to manage resources on your behalf.

1. Open the IAM console.
2. In the navigation menu, choose Roles.
3. Choose Create role.
4. In the Select trusted entity, choose Custom trust policy and paste the following trust policy JSON:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "",
         "Effect": "Allow",
         "Principal": {
           "Service": "ecs-tasks.amazonaws.com"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }

   
5. Click on Next and add the AmazonECSTaskExecutionRolePolicy.
6. Then, for Role Name, enter ecsTaskExecutionRole
7. Finally, click on Create role.

Make sure to copy the ARN of the role. You will need it later.

Create the ecsTaskRole IAM role

On top of the Execution Role, we will also need a Task Role that includes S3 access permissions to store files.

First, we’ll need to create a policy the role can use for accessing S3.

1. Open the IAM console.
2. In the navigation menu, choose Policies.
3. Select JSON and paste the following into the Policy editor:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Action": [
           "s3:GetObject",
           "s3:PutObject",
           "s3:DeleteObject",
           "s3:ListBucket"
         ],
         "Effect": "Allow",
         "Resource": "*"
       }
     ]
   }

   
4. Select Next and type in a name for the policy, such as ecsTaskRoleS3Policy.
5. Once you’re done, select Create policy.

Now create a new role with the same trust policy as above. Attach the new policy before completing.

1. Open the IAM console.
2. In the navigation menu, choose Roles.
3. Choose Create role.
4. In the Select trusted entity, choose Custom trust policy and paste the following trust policy JSON:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "",
         "Effect": "Allow",
         "Principal": {
           "Service": "ecs-tasks.amazonaws.com"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }

5. Click on Next
6. Search for the new policy and check the box on the left. Once you’ve done this, select Next.
7. Then, for Role Name, enter ecsTaskRole
8. Finally, click on Create role.

AWS Batch setup

Go to the AWS Batch console.

Then, click on Get Started. If you don’t see the Get Started button, add #firstRun to the URL:

Follow the wizard to create a new compute environment.

You should see the following text recommending the use of Fargate:

“We recommend using Fargate in most scenarios. Fargate launches and scales the compute to closely match the resource requirements that you specify for the container. With Fargate, you don’t need to over-provision or pay for additional servers. You also don’t need to worry about the specifics of infrastructure-related parameters such as instance type. When the compute environment needs to be scaled up, jobs that run on Fargate resources can get started more quickly. Typically, it takes a few minutes to spin up a new Amazon EC2 instance. However, jobs that run on Fargate can be provisioned in about 30 seconds. The exact time required depends on several factors, including container image size and number of jobs. Learn more.”

We will follow that advice and use Fargate for this tutorial.

Step 1: Select Orchestration type

Select Fargate and click on Next.

Step 2: Create a compute environment

Add a name for your compute environment — here, we chose kestra. You can keep the default settings for everything. Select the VPC and subnets you want to use — you can use the default VPC and subnets and the default VPC security group. Then, click on Next.

Step 3: Create a job queue

Now we can create a job queue. Here, we also name it kestra. You can keep the default settings. Then, click on Next:

Step 4: Create a job definition

Finally, create a job definition. Here, we name it also kestra. Under Execution role, select the role we created earlier (ecsTaskExecutionRole). Besides that, you can keep default settings for everything else (we adjusted the image to ghcr.io/kestra-io/pydata:latest but that’s totally optional). Then, click on Next:

Step 5: Create a job

Finally, create a job named kestra. Click Next to review settings:

Step 6: Review and create

Review your settings and click on Create resources:

Once you see this message, you are all set:

Copy and apply the ARN to your Kestra configuration

Copy the ARN of the compute environment and job queue. You will need to add these to your Kestra configuration.

Create an S3 Bucket

Create an S3 bucket to store input and output files. To do this, open S3 → Create bucket.

Next you’ll need to add a name and leave everything else as a default value.

Scroll to the bottom and select Create bucket.

Now that we have a bucket, we’ll need to add the name into Kestra.

Run your Kestra task on AWS ECS Fargate

Fill in the ARNs of the compute environment and job queue in your Kestra configuration. Here is an example of a flow that uses the aws.runner.Batch to run a Python script on AWS ECS Fargate to get environment information and print it to the logs:

id: aws_batch_runner
namespace: company.team

variables:
  compute_environment_arn: arn:aws:batch:us-east-1:123456789:compute-environment/kestra
  job_queue_arn: arn:aws:batch:us-east-1:123456789:job-queue/kestra
  execution_role_arn: arn:aws:iam::123456789:role/ecsTaskExecutionRole
  task_role_arn: arn:aws:iam::123456789:role/ecsTaskRole

tasks:
  - id: send_data
    type: io.kestra.plugin.scripts.python.Script
    containerImage: ghcr.io/kestra-io/pydata:latest
    taskRunner:
      type: io.kestra.plugin.ee.aws.runner.Batch
      region: us-east-1
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
      computeEnvironmentArn: "{{ vars.compute_environment_arn }}"
      jobQueueArn: "{{ vars.job_queue_arn }}"
      executionRoleArn: "{{ vars.execution_role_arn }}"
      taskRoleArn: "{{ vars.task_role_arn }}"
      bucket: kestra-us
    script: |
      import platform
      import socket
      import sys

      print("Hello from AWS Batch and kestra!")

      def print_environment_info():
          print(f"Host's network name: {platform.node()}")
          print(f"Python version: {platform.python_version()}")
          print(f"Platform information (instance type): {platform.platform()}")
          print(f"OS/Arch: {sys.platform}/{platform.machine()}")

          try:
              hostname = socket.gethostname()
              ip_address = socket.gethostbyname(hostname)
              print(f"Host IP Address: {ip_address}")
          except socket.error as e:
              print("Unable to obtain IP address.")


      if __name__ == '__main__':
          print_environment_info()

When you execute this task, the environment information appears in the logs generated by the Python script:

Advanced configuration

The task runner exposes several optional properties for tuning behavior and authentication.

Polling and timeouts

Job lifecycle

STS role assumption

Instead of static accessKeyId / secretKeyId credentials, you can authenticate via AWS STS AssumeRole for cross-account access or short-lived credentials:

taskRunner:
  type: io.kestra.plugin.ee.aws.runner.Batch
  region: eu-central-1
  stsRoleArn: "arn:aws:iam::123456789012:role/kestra-batch-role"
  stsRoleExternalId: "{{ secret('STS_EXTERNAL_ID') }}"
  stsRoleSessionName: kestra-session
  computeEnvironmentArn: "arn:aws:batch:eu-central-1:123456789012:compute-environment/kestraFargateEnvironment"