AWS Ec2

AWS Ec2

Certified
Enterprise Edition

Task runner that executes a task natively on an AWS EC2 instance

Enterprise Edition only.

This task runner launches an EC2 instance, runs your task's commands directly on it via AWS Systems Manager Run Command (no SSH, no container), then terminates the instance. Reach for it when your work must run natively on a machine, such as native binaries or licensed software tied to a specific AMI. The containerImage property is ignored.

Requirements: the AMI must have the SSM Agent installed (Amazon Linux and recent Ubuntu AMIs have it), and iamInstanceProfile must grant SSM access. To pass inputFiles, outputFiles, or namespaceFiles, set the bucket property, which the runner uses to stage files. Access the working directory with {{ workingDir }} or the WORKING_DIR environment variable.

For a completed command, the task's exit code is the real exit code of your last command. Other terminal SSM statuses (e.g. cancelled or timed out) map to a fixed code instead. Command output appears in the Kestra logs. If the Worker restarts mid-run, resume (default true) reattaches to the same instance instead of starting a new one. The instance is always terminated when the run ends, whether it succeeds, fails, or is killed.

See the plugin documentation for the full IAM, networking, and logging setup.

yaml
type: io.kestra.plugin.ee.aws.runner.Ec2

Launch an EC2 instance and run a Shell command on it.

yaml
id: ec2_shell
namespace: company.team

variables:
  region: eu-west-3
  amiId: ami-0c55b159cbfafe1f0

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.aws.runner.Ec2
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
      region: "{{ vars.region }}"
      amiId: "{{ vars.amiId }}"
      instanceType: t3.micro
      iamInstanceProfile: kestra-ec2-ssm-profile
    commands:
      - echo "Hello World"

Pass input files to the task, execute a Shell command, then retrieve the output files.

yaml
id: ec2_with_input_files
namespace: company.team

inputs:
  - id: file
    type: FILE

variables:
  region: eu-west-3
  amiId: ami-0c55b159cbfafe1f0
  bucket: my-kestra-staging-bucket

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    inputFiles:
      data.txt: "{{ inputs.file }}"
    outputFiles:
      - out.txt
    taskRunner:
      type: io.kestra.plugin.ee.aws.runner.Ec2
      accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
      secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
      region: "{{ vars.region }}"
      amiId: "{{ vars.amiId }}"
      instanceType: t3.micro
      iamInstanceProfile: kestra-ec2-ssm-profile
      subnetId: subnet-0123456789abcdef0
      securityGroupIds:
        - sg-0123456789abcdef0
      bucket: "{{ vars.bucket }}"
    commands:
      - cp {{ workingDir }}/data.txt {{ workingDir }}/out.txt
Properties

AMI ID to launch the instance from

The ID of the Amazon Machine Image used to launch the instance, for example ami-0c55b159cbfafe1f0. The AMI must have the SSM Agent installed and running (true by default for modern Amazon Linux 2/2023 and Ubuntu AMIs), since this task runner executes the task via AWS Systems Manager Run Command rather than SSH.

IAM instance profile

Name or ARN of the IAM instance profile attached to the instance. Must grant at minimum the AmazonSSMManagedInstanceCore managed policy (or equivalent) so the SSM Agent can register with Systems Manager and receive commands, plus S3 read/write access to bucket if file staging is used. AmazonSSMManagedInstanceCore alone does not grant CloudWatch Logs permissions: if streamLogs is true (the default), also grant logs: CreateLogGroup, logs: CreateLogStream, and logs: PutLogEvents.

EC2 instance type

The compute capacity of the instance, for example t3.micro or m5.large. See the AWS documentation for available types.

AWS region with which the SDK should communicate

Access Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

S3 Bucket to upload (inputFiles and namespaceFiles) and download (outputFiles) files

It's mandatory to provide a bucket if you want to use such properties.

DefaultPT5S

Determines how often Kestra should poll the SSM command for completion. By default, the task runner checks every 5 seconds whether the command is completed. You can set this to a lower value (e.g. PT0.1S = every 100 milliseconds) for quick tasks and to a higher threshold (e.g. PT1M = every minute) for long-running tasks. Setting this property to a lower value will reduce the number of API calls Kestra makes to the remote service, so keep that in mind in case you see API rate limit errors

The endpoint with which the SDK should communicate

This property allows you to use a different S3 compatible storage backend.

DefaultPT5M

Instance and SSM Agent readiness timeout

The maximum duration to wait for the launched instance to reach the running state and for its SSM Agent to register with Systems Manager, before failing the task. Defaults to PT5M. Heavy or custom AMIs (e.g. ones that run lengthy boot-time provisioning) may need a higher value.

Reference (ref) of the pluginDefaults to apply to this task runner.

Defaulttrue

Whether to reconnect to the current instance and command if they already exist

Secret Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

SubTypestring

Security groups to attach to the instance

List of security group IDs to associate with the instance's network interface. If not set, the subnet's default security group is used.

AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

Maximum Spot price per hour

If set, the instance is launched as a Spot Instance with this maximum hourly price in USD (e.g. 0.05). Leave unset to launch a standard On-Demand instance. Spot instances can be reclaimed by AWS at any time. Only use this for interruption-tolerant workloads.

Defaulttrue

Whether to poll CloudWatch Logs for near real-time command output

When true (default), the task runner polls the /aws/ec2/kestra-run-command CloudWatch log group for this command's output while it runs, so it appears in the Kestra execution as it is produced. This requires the instance's iamInstanceProfile to grant logs: CreateLogGroup, logs: CreateLogStream, and logs: PutLogEvents (AmazonSSMManagedInstanceCore alone does not include these). Set this to false to skip that live polling entirely: no CloudWatch API calls are made. Either way, command output always reaches Kestra once the command finishes: the task runner falls back to reading StandardOutputContent/StandardErrorContent straight from SSM's GetCommandInvocation response (capped at roughly 24 KB per stream) whenever nothing was already streamed from CloudWatch. Command polling and exit-code mapping are unaffected by this property either way.

The AWS STS endpoint with which the SDKClient should communicate

AWS STS Role

The Amazon Resource Name (ARN) of the role to assume. If set the task will use the StsAssumeRoleCredentialsProvider. If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

AWS STS External Id

A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.

DefaultPT15M

AWS STS Session duration

The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.

AWS STS Session name

This property is only used when an stsRoleArn is defined.

Subnet to launch the instance in

The ID of the subnet the instance's network interface is attached to. If not set, AWS launches the instance into your account's default VPC/subnet. The instance must be able to reach the AWS Systems Manager and (if bucket is set) S3 endpoints, either via a NAT/internet gateway or VPC endpoints.

Defaultfalse

Whether to synchronize working directory from remote runner back to local one after run.

Plugin Version

Defines the version of the plugin to use.

The version must follow the Semantic Versioning (SemVer) specification:

  • A single-digit MAJOR version (e.g., 1).
  • A MAJOR.MINOR version (e.g., 1.1).
  • A MAJOR.MINOR.PATCH version, optionally with any qualifier (e.g., 1.1.2, 1.1.0-SNAPSHOT).
DefaultPT5S

Post-completion CloudWatch log drain period

Once the command reaches a terminal status, the task runner keeps polling CloudWatch Logs until no new output has arrived for this long, so a short command's entire output burst, which can land in CloudWatch a second or two after the command already finished, isn't lost. Only used when streamLogs is true. Defaults to PT5S.

DefaultPT1H

The maximum duration to wait for the SSM command completion unless the task timeout property is set which will take precedence over this property

The task will be marked as failed if the command has not completed by the time this duration elapses.

More Plugins in this Category

Core Plugins and tasks
Core Plugins and tasks

Tasks that provide Kestra's built-in orchestration, I/O, and observability capabilities.

Core

122 tasks

398 blueprints

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks