How to use the AWS EE plugin

Run tasks on AWS Batch or natively on EC2, ship execution logs to CloudWatch and S3, and orchestrate AWS Batch jobs from Kestra flows.

Authentication

All tasks support three authentication methods. Set accessKeyId and secretKeyId for static credentials, optionally adding sessionToken for temporary credentials. Set stsRoleArn to assume an IAM role via STS (optionally with stsRoleExternalId, stsRoleSessionName, stsRoleSessionDuration default PT15M). If none are set, the AWS default credentials provider chain is used (environment variables, instance metadata, etc.). Store secrets in secrets and apply connection properties globally with plugin defaults.

Tasks

batch.Run submits and monitors an existing AWS Batch job. Set region, jobQueue, and jobDefinition (all required). Optionally set jobName (default kestra-{execution.id}), parameters, containerCommand, containerEnvironment, wait (default true), maxWait (default PT6H), completionCheckInterval (default PT10S), completionCheckMaxInterval (default PT1M), streamLogs (default false), and logGroupName (default /aws/batch/job). Outputs include jobId, jobName, jobQueue, jobDefinition, status, statusReason, createdAt, startedAt, and stoppedAt.

The task role associated with the job needs batch: SubmitJob, batch: DescribeJobs, batch: ListJobs, batch: TagResource, batch: RegisterJobDefinition, batch: DeregisterJobDefinition, batch: DescribeComputeEnvironments, logs: DescribeLogGroups, logs: StartLiveTail, and iam: PassRole.

Task runners

runner.Batch runs task scripts inside AWS Batch jobs on ECS (Fargate or EC2) or EKS compute environments. Set region and computeEnvironmentArn (both required). Optionally set jobQueueArn (auto-created if omitted, but adds latency), executionRoleArn (required for Fargate), taskRoleArn (for container IAM permissions on ECS), serviceAccountName (for EKS IRSA), and bucket (S3 bucket for staging inputFiles, outputFiles, and namespaceFiles).

Configure compute resources via resources.request with memory (MiB) and cpu (vCPUs, Fargate accepts 0.25–16, EC2 requires integers). Sidecar container resources are controlled by sidecarResources.request.

Key behavioral properties: delete (default true), resume (default true), waitUntilCompletion (default PT1H), completionCheckInterval (default PT5S), streamLogs (default true).

Tasks run in a multi-container job: a before container uploads input files to S3, the main container fetches them and executes the task, and an after container captures output files. Access the working directory via {{ workingDir }} or the WORKING_DIR environment variable.

EKS compute environments

To set up an EKS cluster for use with AWS Batch, follow the cluster setup walkthrough in the Kestra documentation. The walkthrough covers cluster creation, RBAC, IAM identity mapping, the amazon-cloudwatch-observability addon, and compute environment and job queue creation.

Set serviceAccountName on the task runner if the pod needs IAM access via IRSA.

runner.Ec2 runs task scripts natively on a plain EC2 instance via AWS Systems Manager Run Command. No container is involved, so containerImage is ignored. Set region, amiId, instanceType, and iamInstanceProfile (all required). The AMI must have the SSM Agent installed and running (true by default for modern Amazon Linux 2/2023 and Ubuntu AMIs), and iamInstanceProfile must grant at least AmazonSSMManagedInstanceCore. Optionally set subnetId, securityGroupIds, spotMaxPrice (to launch a Spot Instance), and bucket (S3 bucket for staging inputFiles, outputFiles, and namespaceFiles, which requires the AWS CLI to be available on the AMI).

Key behavioral properties: resume (default true), waitUntilCompletion (default PT1H), completionCheckInterval (default PT5S), streamLogs (default true), waitForLogInterval (default PT5S), instanceReadyTimeout (default PT5M, raise it for heavy or custom AMIs that need longer to boot and register their SSM Agent). For a COMPLETED command, the exit code is the real exit code of the last command executed. Other terminal SSM statuses (e.g. cancelled or timed out) map to a fixed, synthetic exit code instead. If the instance reaches a terminal state (e.g. TERMINATED or STOPPED, such as after a Spot reclaim) before becoming ready, the task fails immediately with the instance's own state reason instead of waiting out the full timeout. The instance is always terminated once the run completes, whether successfully, on failure, or on kill, so it never leaves a zombie instance running (termination itself is retried a few times before giving up). On a Worker restart, resume reattaches to the same instance and in-flight command instead of creating a duplicate.

Command output always reaches Kestra reliably, even without CloudWatch: once the command finishes, the task runner reads the standard output/error captured directly by SSM's own GetCommandInvocation response and logs it if nothing was already streamed. When streamLogs is true (the default), the task runner also polls the /aws/ec2/kestra-run-command CloudWatch log group for near-real-time output while the command runs, and keeps draining it for waitForLogInterval after the command finishes so a short command's entire output burst, which can land in CloudWatch a second or two after completion, isn't lost. AmazonSSMManagedInstanceCore does not grant CloudWatch Logs permissions: if streamLogs is true, also grant the instance role logs: CreateLogGroup, logs: CreateLogStream, and logs: PutLogEvents, or set streamLogs to false to rely on the GetCommandInvocation fallback alone (SSM caps that fallback at roughly 24 KB per stream).

Environment variables are embedded, quoted, in the SSM command document sent to the instance. Any AWS principal with ssm: GetCommandInvocation or ssm: ListCommands permissions in your account can read them back. Avoid passing long-lived secrets through environment variables on this task runner, prefer short-lived credentials instead.

Log exporters

cloudwatch.LogExporter ships execution logs to AWS CloudWatch Logs. Set region, logGroupName, and logStreamName (all required). Optionally set chunk (logs per request, default 1000).

s3.LogExporter ships execution logs to AWS S3. Set region and bucket (both required). Optionally set format (default JSON, also accepts ION), maxLinesPerFile (default 100000), logFilePrefix (default kestra-log-file), and chunk (default 1000).