AwsBatchTaskRunner
AwsBatchTaskRunner
This plugin is currently in beta. While it is considered safe for use, please be aware that its API could change in ways that are not compatible with earlier versions in future releases, or it might become unsupported.
type: "io.kestra.plugin.aws.runner.AwsBatchTaskRunner"
Task runner that executes a task inside a job in AWS Batch.
This task runner only supports ECS Fargate or ECS EC2 as compute environment. For EKS, use the KubernetesTaskRunner.
Make sure to set the containerImage
property because this runner runs the task in a container.
To access the task's working directory, use the Pebble expression or the
WORKING_DIR
environment variable.
This directory will contain all input files and namespace files (if enabled).
To generate output files you can either use the outputFiles
task property and create a file with the same name in the task's working directory, or create any file in the output directory which can be accessed using the Pebble expression or the
OUTPUT_DIR
environment variable.
To use inputFiles
, outputFiles
or namespaceFiles
properties, make sure to set the bucket
property. The bucket serves as an intermediary storage layer for the task runner. Input and namespace files will be uploaded to the cloud storage bucket before the task run starts. Similarly, the task runner will store outputFiles
in this bucket during the task run. In the end, the task runner will make those files available for download and preview from the UI by sending them to internal storage.
To make it easier to track where all files are stored, the task runner will generate a folder for each task run. You can access that folder using the Pebble expression or the
BUCKET_PATH
environment variable.
Note that this task runner executes the task in the root directory. You need to use the Pebble expression or the
WORKING_DIR
environment variable to access files in the task's working directory.
This task runner will return with an exit code according to the following mapping:
- SUCCEEDED: 0
- FAILED: 1
- RUNNING: 2
- RUNNABLE: 3
- PENDING: 4
- STARTING: 5
- SUBMITTED: 6
- OTHER: -1
When the Kestra Worker running this task is terminated, the batch job will still run until completion.
To avoid zombie containers in ECS, you can set the timeout
property on the task and kestra will terminate the batch job if the task is not completed within the specified duration.
Examples
Execute a Shell command in a container on ECS Fargate.
id: run_container
namespace: dev
tasks:
- id: shell
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.aws.runner.AwsBatchTaskRunner
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
region: "{{ vars.region }}"
computeEnvironmentArn: "{{ vars.computeEnvironmentArn }}"
commands:
- echo "Hello World"
Pass input files to the task, execute a Shell command, then retrieve the output files.
id: container_with_input_files
namespace: myteam
inputs:
- id: file
type: FILE
tasks:
- id: shell
type: io.kestra.plugin.scripts.shell.Commands
inputFiles:
data.txt: "{{ inputs.file }}"
outputFiles:
- out.txt
containerImage: centos
taskRunner:
type: io.kestra.plugin.aws.runner.AwsBatchTaskRunner
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
region: "{{ vars.region }}"
computeEnvironmentArn: "{{ vars.computeEnvironmentArn }}"
bucket: "{{ vars.bucket }}"
commands:
- cp {{ workingDir }}/data.txt {{ workingDir }}/out.txt
Properties
computeEnvironmentArn
- Type: string
- Dynamic: ✔️
- Required: ✔️
Compute environment in which to run the job.
region
- Type: string
- Dynamic: ✔️
- Required: ✔️
AWS region with which the SDK should communicate.
resources
- Type: AwsBatchTaskRunner-Resources
- Dynamic: ❌
- Required: ✔️
- Default:
{request={memory=2048, cpu=1}}
Custom resources for the ECS Fargate container.
See the AWS documentation for more details.
accessKeyId
- Type: string
- Dynamic: ✔️
- Required: ❌
Access Key Id in order to connect to AWS.
If no connection is defined, we will use the
DefaultCredentialsProvider
to fetch the value.
bucket
- Type: string
- Dynamic: ✔️
- Required: ❌
S3 Bucket to upload (inputFiles
and namespaceFiles
) and download (outputFiles
) files.
It's mandatory to provide a bucket if you want to use such properties.
endpointOverride
- Type: string
- Dynamic: ✔️
- Required: ❌
The endpoint with which the SDK should communicate.
This property should normally not be used except for local development.
executionRoleArn
- Type: string
- Dynamic: ✔️
- Required: ❌
Execution role for the AWS Batch job.
Mandatory if the compute environment is ECS Fargate. See the AWS documentation for more details.
jobQueueArn
- Type: string
- Dynamic: ✔️
- Required: ❌
Job queue to use to submit jobs (ARN). If not specified, the task runner will create a job queue — keep in mind that this can lead to a longer execution.
secretKeyId
- Type: string
- Dynamic: ✔️
- Required: ❌
Secret Key Id in order to connect to AWS.
If no connection is defined, we will use the
DefaultCredentialsProvider
to fetch the value.
sessionToken
- Type: string
- Dynamic: ✔️
- Required: ❌
AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource.
If no connection is defined, we will use the
DefaultCredentialsProvider
to fetch the value.
stsEndpointOverride
- Type: string
- Dynamic: ✔️
- Required: ❌
The AWS STS endpoint with which the SDKClient should communicate.
stsRoleArn
- Type: string
- Dynamic: ✔️
- Required: ❌
AWS STS Role.
The Amazon Resource Name (ARN) of the role to assume. If set the task will use the
StsAssumeRoleCredentialsProvider
. Otherwise, theStaticCredentialsProvider
will be used with the provided Access Key Id and Secret Key.
stsRoleExternalId
- Type: string
- Dynamic: ✔️
- Required: ❌
AWS STS External Id.
A unique identifier that might be required when you assume a role in another account. This property is only used when an
stsRoleArn
is defined.
stsRoleSessionDuration
- Type: string
- Dynamic: ❌
- Required: ❌
- Default:
900.000000000
- Format:
duration
AWS STS Session duration.
The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an
stsRoleArn
is defined.
stsRoleSessionName
- Type: string
- Dynamic: ✔️
- Required: ❌
AWS STS Session name. This property is only used when an stsRoleArn
is defined.
taskRoleArn
- Type: string
- Dynamic: ✔️
- Required: ❌
Task role to use within the container.
Needed if you want to authenticate with AWS CLI within your container.
waitUntilCompletion
- Type: string
- Dynamic: ❓
- Required: ❌
- Default:
3600.000000000
- Format:
duration
The maximum duration to wait for the job completion unless the task timeout
property is set which will take precedence over this property.
AWS Batch will automatically timeout the job upon reaching that duration and the task will be marked as failed.
Definitions
io.kestra.plugin.aws.runner.AwsBatchTaskRunner-Resource
Properties
cpu
- Type: string
- Dynamic: ❓
- Required: ✔️
memory
- Type: string
- Dynamic: ❓
- Required: ✔️
io.kestra.plugin.aws.runner.AwsBatchTaskRunner-Resources
Properties
request
- Type: AwsBatchTaskRunner-Resource
- Dynamic: ❓
- Required: ✔️