Execute Kestra Tasks Directly on AWS EC2 Instances

Run tasks natively on AWS EC2 instances via AWS Systems Manager Run Command — no SSH, no container runtime required.

When to use the EC2 task runner

Use the EC2 task runner when your workload cannot or should not run inside a container:

• GPU workloads that require a specific CUDA-capable AMI without the overhead of AWS Batch queue setup
• Non-containerized scripts that use locally installed binaries or proprietary software tied to a specific AMI
• Spot instance optimization where you need direct control over the instance type and max bid price
• Network-sensitive workloads that must run inside a specific VPC subnet with fine-grained security group control

For containerized workloads on AWS, use the AWS Batch task runner instead.

How it works

For each task run, the EC2 task runner:

1. Launches an EC2 instance from the configured AMI and instance type
2. Waits for the instance to reach running state and for the SSM Agent to register with Systems Manager (up to instanceReadyTimeout, default 5 minutes)
3. If bucket is set, uploads inputFiles and namespaceFiles to S3 from the Kestra worker
4. Sends the task’s commands to the instance via SSM Run Command, which also downloads input files from S3 and uploads output files back to S3 on the instance side
5. Streams command output from CloudWatch Logs to the Kestra execution logs in near real-time (when streamLogs: true, the default)
6. Downloads outputFiles from S3 to Kestra internal storage once the command completes
7. Terminates the instance unconditionally — on success, failure, or worker interruption

This runner ignores the containerImage property. Commands execute directly on the instance OS.

The following Pebble expressions and environment variables are available inside the task:

Exit codes

When the SSM command reaches a successful or failed terminal state, the task’s exit code is the real exit code of your last command. For other terminal SSM statuses, the runner maps them to fixed codes:

Minimum IAM permissions required

The IAM principal used by Kestra (the user or role whose credentials are passed via accessKeyId/secretKeyId, or the worker’s instance profile) needs the following minimum permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:RunInstances",
        "ec2:DescribeInstances",
        "ec2:CreateTags",
        "ec2:TerminateInstances",
        "ssm:SendCommand",
        "ssm:GetCommandInvocation",
        "ssm:DescribeInstanceInformation",
        "ssm:ListCommands",
        "ssm:CancelCommand",
        "logs:DescribeLogGroups",
        "logs:CreateLogGroup"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:PassRole"
      ],
      "Resource": "<iamInstanceProfileRoleArn>"
    }
  ]
}

Replace <iamInstanceProfileRoleArn> with the ARN of the IAM role backing your instance profile.

Two of these permissions are only exercised in specific situations:

• ssm:ListCommands — used during resume when the instance is found but its command ID tag is missing (fallback to locate the in-flight command)
• ssm:CancelCommand — used when the task is killed to cancel the in-flight SSM command before terminating the instance

CloudWatch Logs streaming permission

When streamLogs is true (the default), the runner polls CloudWatch Logs in real time using logs:FilterLogEvents. Add this permission to the statement above:

{
  "Effect": "Allow",
  "Action": [
    "logs:FilterLogEvents"
  ],
  "Resource": "arn:aws:logs:<region>:<accountId>:log-group:/aws/ec2/kestra-run-command:*"
}

Set streamLogs: false to skip live polling. Command output is always delivered at the end of the run via SSM’s GetCommandInvocation response (capped at ~24 KB per stream) regardless of this setting.

S3 permissions when using bucket

When the bucket property is set, the Kestra worker uploads inputFiles and namespaceFiles to S3 before the instance runs, and downloads outputFiles after the command completes. Add these permissions to the Kestra principal:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": "*"
    }
  ]
}

IAM instance profile requirements

The EC2 instance profile attached via iamInstanceProfile must allow the SSM Agent to register and receive commands. At minimum, attach the AmazonSSMManagedInstanceCore AWS managed policy to the instance profile’s role.

If bucket is set, the instance also needs S3 access to download input files and upload output files at runtime. The instance uses the AWS CLI (aws s3 sync / aws s3 cp) to transfer files, so grant the following on the staging bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<bucket>",
        "arn:aws:s3:::<bucket>/*"
      ]
    }
  ]
}

If streamLogs is true (the default), the instance also needs permission to write logs to CloudWatch. AmazonSSMManagedInstanceCore does not include these. Add them to the instance profile role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "*"
    }
  ]
}

AMI requirements

The AMI must have the SSM Agent installed and running. This is true by default for:

• Amazon Linux 2 and Amazon Linux 2023
• Ubuntu Server 16.04 LTS and later (AWS-provided AMIs)
• Most AWS-provided Windows Server AMIs

For custom AMIs, verify the SSM Agent is present and enabled. See the SSM Agent installation guide for details.

Example flows

Basic shell command

id: ec2_shell
namespace: company.team

variables:
  region: eu-west-3
  amiId: ami-0c55b159cbfafe1f0

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.aws.runner.Ec2
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
      region: "{{ vars.region }}"
      amiId: "{{ vars.amiId }}"
      instanceType: t3.micro
      iamInstanceProfile: kestra-ec2-ssm-profile
    commands:
      - echo "Hello from EC2"

Input and output files via S3

id: ec2_with_files
namespace: company.team

inputs:
  - id: file
    type: FILE

variables:
  region: eu-west-3
  amiId: ami-0c55b159cbfafe1f0
  bucket: my-kestra-staging-bucket

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    inputFiles:
      data.txt: "{{ inputs.file }}"
    outputFiles:
      - out.txt
    taskRunner:
      type: io.kestra.plugin.ee.aws.runner.Ec2
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
      region: "{{ vars.region }}"
      amiId: "{{ vars.amiId }}"
      instanceType: t3.micro
      iamInstanceProfile: kestra-ec2-ssm-profile
      subnetId: subnet-0123456789abcdef0
      securityGroupIds:
        - sg-0123456789abcdef0
      bucket: "{{ vars.bucket }}"
    commands:
      - cp "{{ workingDir }}/data.txt" "{{ workingDir }}/out.txt"

GPU workload on a Spot instance

id: gpu_spot_job
namespace: company.team

tasks:
  - id: inference
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.aws.runner.Ec2
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
      region: "{{ secret('AWS_REGION') }}"
      amiId: "{{ secret('DEEP_LEARNING_AMI_ID') }}"
      instanceType: g4dn.xlarge
      subnetId: "{{ secret('PRIVATE_SUBNET_ID') }}"
      securityGroupIds:
        - "{{ secret('SG_ID') }}"
      iamInstanceProfile: "{{ secret('INSTANCE_PROFILE_ARN') }}"
      bucket: "{{ secret('S3_BUCKET') }}"
      spotMaxPrice: "0.25"
    commands:
      - python /opt/myapp/inference.py --model /opt/models/my-model

Key properties

For the full property reference, see the plugin documentation.