How to use the AWS EE plugin

Run tasks on AWS Batch, natively on EC2, or in AWS Lambda MicroVMs, ship execution logs to CloudWatch and S3, and orchestrate AWS Batch jobs from Kestra flows.

Authentication

All tasks support three authentication methods. Set accessKeyId and secretKeyId for static credentials, optionally adding sessionToken for temporary credentials. Set stsRoleArn to assume an IAM role via STS (optionally with stsRoleExternalId, stsRoleSessionName, stsRoleSessionDuration default PT15M). If none are set, the AWS default credentials provider chain is used (environment variables, instance metadata, etc.). Store secrets in secrets and set connection properties on each task.

Tasks

batch.Run submits and monitors an existing AWS Batch job. Set region, jobQueue, and jobDefinition (all required). Optionally set jobName (default kestra-{execution.id}), parameters, containerCommand, containerEnvironment, wait (default true), maxWait (default PT6H), completionCheckInterval (default PT10S), completionCheckMaxInterval (default PT1M), streamLogs (default false), logGroupName (default /aws/batch/job), and failOnFailedStatus (default true; set false to let the task succeed even when the Batch job ends with status FAILED). Outputs include jobId, jobName, jobQueue, jobDefinition, status, statusReason, createdAt, startedAt, and stoppedAt.

The task role associated with the job needs batch: SubmitJob, batch: DescribeJobs, batch: ListJobs, batch: TagResource, batch: RegisterJobDefinition, batch: DeregisterJobDefinition, batch: DescribeComputeEnvironments, logs: DescribeLogGroups, logs: StartLiveTail, and iam: PassRole.

Task runners

runner.Batch runs task scripts inside AWS Batch jobs on ECS (Fargate or EC2) or EKS compute environments. Set region and computeEnvironmentArn (both required). Optionally set jobQueueArn (auto-created if omitted, but adds latency), executionRoleArn (required for Fargate), taskRoleArn (for container IAM permissions on ECS), serviceAccountName (for EKS IRSA), and bucket (S3 bucket for staging inputFiles, outputFiles, and namespaceFiles).

Configure compute resources via resources.request with memory (MiB) and cpu (vCPUs, Fargate accepts 0.25–16, EC2 requires integers). Sidecar container resources are controlled by sidecarResources.request.

Key behavioral properties: delete (default true), resume (default true), waitUntilCompletion (default PT1H), completionCheckInterval (default PT5S), streamLogs (default true).

Tasks run in a multi-container job: a before container uploads input files to S3, the main container fetches them and executes the task, and an after container captures output files. Access the working directory via {{ workingDir }} or the WORKING_DIR environment variable.

monitoring.enabled: true runs the command under kotlp and reports CPU, memory and disk I/O as task metrics, with stderr logged at ERROR. Requires bucket, and an image with /bin/sh, gzip and a writable $TMPDIR (default /tmp).

EKS compute environments

To set up an EKS cluster for use with AWS Batch, follow the cluster setup walkthrough in the Kestra documentation. The walkthrough covers cluster creation, RBAC, IAM identity mapping, the amazon-cloudwatch-observability addon, and compute environment and job queue creation.

Set serviceAccountName on the task runner if the pod needs IAM access via IRSA.

runner.Ec2 runs task scripts natively on a plain EC2 instance via AWS Systems Manager Run Command. No container is involved, so containerImage is ignored. Set region, amiId, instanceType, and iamInstanceProfile (all required). The AMI must have the SSM Agent installed and running (true by default for modern Amazon Linux 2/2023 and Ubuntu AMIs), and iamInstanceProfile must grant at least AmazonSSMManagedInstanceCore. Optionally set subnetId, securityGroupIds, associatePublicIp (whether the instance gets a public IP; requires subnetId), requireImdsv2 (enforce IMDSv2 session tokens on the instance metadata service; only the token requirement is set, so the hop limit stays at the default of 1 and containers started by your task, e.g. via docker run, cannot fetch instance profile credentials), spotMaxPrice (to launch a Spot Instance), and bucket (S3 bucket for staging inputFiles, outputFiles, and namespaceFiles, which requires the AWS CLI to be available on the AMI). For a private subnet without a public IP, SSM, S3 (when bucket is set) and, when streamLogs is true, CloudWatch Logs must still be reachable through a NAT gateway or VPC endpoints, or set streamLogs to false. Secure Boot is obtained by choosing a UEFI Secure Boot-enabled AMI via amiId.

Key behavioral properties: resume (default true), waitUntilCompletion (default PT1H), completionCheckInterval (default PT5S), streamLogs (default true), waitForLogInterval (default PT5S), instanceReadyTimeout (default PT5M, raise it for heavy or custom AMIs that need longer to boot and register their SSM Agent). For a COMPLETED command, the exit code is the real exit code of the last command executed. Other terminal SSM statuses (e.g. cancelled or timed out) map to a fixed, synthetic exit code instead. If the instance reaches a terminal state (e.g. TERMINATED or STOPPED, such as after a Spot reclaim) before becoming ready, the task fails immediately with the instance's own state reason instead of waiting out the full timeout. The instance is always terminated once the run completes, whether successfully, on failure, or on kill, so it never leaves a zombie instance running (termination itself is retried a few times before giving up). On a Worker restart, resume reattaches to the same instance and in-flight command instead of creating a duplicate.

Command output always reaches Kestra reliably, even without CloudWatch: once the command finishes, the task runner reads the standard output/error captured directly by SSM's own GetCommandInvocation response and logs it if nothing was already streamed. When streamLogs is true (the default), the task runner also polls the /aws/ec2/kestra-run-command CloudWatch log group for near-real-time output while the command runs, and keeps draining it for waitForLogInterval after the command finishes so a short command's entire output burst, which can land in CloudWatch a second or two after completion, isn't lost. AmazonSSMManagedInstanceCore does not grant CloudWatch Logs permissions: if streamLogs is true, also grant the instance role logs: CreateLogGroup, logs: CreateLogStream, and logs: PutLogEvents, or set streamLogs to false to rely on the GetCommandInvocation fallback alone (SSM caps that fallback at roughly 24 KB per stream).

Environment variables are embedded, quoted, in the SSM command document sent to the instance. Any AWS principal with ssm: GetCommandInvocation or ssm: ListCommands permissions in your account can read them back. Avoid passing long-lived secrets through environment variables on this task runner, prefer short-lived credentials instead.

monitoring.enabled: true runs the command under kotlp and reports CPU, memory and disk I/O as task metrics, with stderr logged at ERROR. Requires bucket.

runner.LambdaMicroVm runs task scripts in an AWS Lambda MicroVM. containerImage is ignored. Set region, bucket, buildRoleArn (IAM role Lambda assumes to build the image) and executionRoleArn (IAM role the microVM runs as); nothing else is required. The first run builds a microVM image holding the Kestra agent in your account (one time, a few minutes; built from a Dockerfile on public.ecr.aws/lambda/microvms: al2023-minimal, so commands get the Amazon Linux 2023 minimal userland, arm64), named kestra-task-runner-<hash> and reused by later runs. A new plugin version, another baseImageArn or another memory builds a separate image. If the build fails, the error carries AWS's reason and the build log location: fix the cause, delete the failed image and rerun. To skip the build, set imageIdentifier (and imageVersion) to your own image, which must embed the Kestra agent (the bootstrap executable, listening on 0.0.0.0: 8080 and serving POST /aws/lambda-microvms/runtime/v1/{run,terminate,validate,ready}); buildRoleArn is then not needed, but executionRoleArn still is.

Optionally set memory (MiB, applies to the image), logGroup (default /aws/lambda-microvm/kestra), baseImageArn (defaults to the only managed base image AWS lists for the region, otherwise it must be set), ingressNetworkConnectors, egressNetworkConnectors, idlePolicy (maxIdleDuration, suspendedDuration and autoResumeEnabled, sent to AWS only when set; a suspended microVM stops running your commands) and tags (applied to the image). Key behavioral properties: waitUntilCompletion (default PT1H, at most PT8H: a longer value is rejected, never shortened) completionCheckInterval (default PT5S) and waitForLogInterval (default PT5S, how long the log tail keeps draining CloudWatch after the task ends).

Files and results travel through bucket under kestra-lambda-microvm/: the microVM reads and writes them through presigned links, so it needs no IAM permission for that. The runner's credentials must outlive the run because they sign the links the microVM writes its result with: with temporary credentials, the run fails fast if they expire first. With stsRoleArn, the session defaults to the wait of the run (waitUntilCompletion, or the task timeout when it is set) plus 15 minutes, capped at 1 hour (the default maximum session of a role). A wait above one hour is rejected up front unless you set stsRoleSessionDuration, which the role's maximum session duration must allow. The links to the bundle and the input files only live 15 minutes. The staged objects, including the task's environment variables, are deleted when the run ends (the agent's code artifact stays and is reused). The bundle, the input archive and the agent's code artifact are uploaded with the encryption set by serverSideEncryption (default AES256, or aws: kms / aws: kms: dsse with an optional kmsKeyId); output.zip and status.json, written by the microVM through presigned links, use the bucket's default encryption. With KMS, the runner credentials need kms: GenerateDataKey and kms: Decrypt on the key and the build role needs kms: Decrypt. As a safety net for runs killed before they could clean up, add an S3 lifecycle rule that expires kestra-lambda-microvm/run/ after one day. Command output is read from the CloudWatch log group while the run goes on, and the exit code is the real exit code of the commands. The microVM is always terminated when the run ends, whether it succeeds, fails or is killed, and termination is retried a few times.

Differences from Batch and Ec2: no containerImage, no vCPU or disk setting (the Lambda MicroVMs API only exposes memory, so there is nothing to configure), microVMs are arm64 only (binaries and native Python wheels must be arm64), a run lasts at most 8 hours, and the first run in a region builds the image. A microVM cannot be tagged, so tags only applies to the image. On a successful run the runtime details (microvmId, state, stateReason, startedAt, stoppedAt, imageArn, imageVersion, managedImage, logGroup, logStream, region) are in {{ outputs.<taskId>.taskRunner }}. Kestra core drops the taskRunner outputs when a task fails, so on a failed run, once a microVM was requested, the same fields are in {{ outputs.<taskId>.vars.kestraTaskRunner }}. This is a workaround until core can carry the details on failure.

The credentials of the runner need the Lambda MicroVM actions (lambda: RunMicrovm, lambda: GetMicrovm, lambda: TerminateMicrovm, lambda: CreateMicrovmImage, lambda: GetMicrovmImage, lambda: ListMicrovmImages, lambda: GetMicrovmImageVersion, lambda: ListManagedMicrovmImages, lambda: ListManagedMicrovmImageVersions, lambda: TagResource), iam: PassRole on buildRoleArn and executionRoleArn (with the condition iam: PassedToService equal to lambda.amazonaws.com), lambda: PassNetworkConnector on arn: aws: lambda: *: aws: network-connector: * (with no connector set, AWS applies its managed INTERNET_EGRESS and HTTP_INGRESS connectors, and passing them needs this permission; add the ARNs of any custom connector set in ingressNetworkConnectors or egressNetworkConnectors), s3: PutObject, s3: GetObject, s3: DeleteObject and s3: AbortMultipartUpload on arn: aws: s3: : : <bucket>/kestra-lambda-microvm/* plus s3: ListBucket and s3: GetBucketLocation on the bucket (arn: aws: s3: : : <bucket>), logs: DescribeLogGroups (on all resources), and logs: CreateLogGroup and logs: FilterLogEvents on the log group ARN (arn: aws: logs: <region>: <account>: log-group: <logGroup>: *). The action names follow the API operation names.

The role in buildRoleArn is assumed by Lambda to build the image. Its trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {"Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts: AssumeRole"}
  ]
}

Its permissions policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {"Effect": "Allow", "Action": "s3: GetObject", "Resource": "arn: aws: s3: : : <bucket>/kestra-lambda-microvm/bootstrap/*"},
    {"Effect": "Allow", "Action": ["logs: CreateLogStream", "logs: PutLogEvents"], "Resource": "arn: aws: logs: <region>: <account>: log-group: <logGroup>: *"}
  ]
}

The role in executionRoleArn is required, also with a custom imageIdentifier: the runner has the microVM write its logs to CloudWatch, reads the command output back from them, and AWS writes those logs as this role. It uses the same trust policy (lambda.amazonaws.com) and needs the log permissions below, plus whatever your own commands need (for example s3: GetObject for aws s3 cp):

{
  "Version": "2012-10-17",
  "Statement": [
    {"Effect": "Allow", "Action": ["logs: CreateLogStream", "logs: PutLogEvents"], "Resource": "arn: aws: logs: <region>: <account>: log-group: <logGroup>: *"}
  ]
}

For a quick setup, one role can serve as both the build and the execution role (same trust policy, merged permissions); separate roles are cleaner. In the IAM console, create the roles with the trusted entity "AWS service" and the use case "Lambda".

The agent's hooks are not authenticated. With no ingressNetworkConnectors set, AWS attaches its managed HTTP_INGRESS connector to the microVM. According to the AWS samples, reaching a microVM through that ingress requires a MicroVM auth token (lambda: CreateMicrovmAuthToken), so keep that permission restricted to trusted principals. Do not add other ingress connectors that would expose port 8080: any peer that can reach it could start a job or stop the running one.

monitoring.enabled: true runs the command under kotlp and reports CPU, memory and disk I/O as task metrics, with stderr logged at ERROR. Requires commands.

Log exporters

cloudwatch.LogExporter ships execution logs to AWS CloudWatch Logs. Set region, logGroupName, and logStreamName (all required). Optionally set chunk (logs per request, default 1000).

s3.LogExporter ships execution logs to AWS S3. Set region and bucket (both required). Optionally set format (default JSON, also accepts ION), maxLinesPerFile (default 100000), logFilePrefix (default kestra-log-file), and chunk (default 1000).