AzureBatchTaskRunner
AzureBatchTaskRunner
This plugin is currently in beta. While it is considered safe for use, please be aware that its API could change in ways that are not compatible with earlier versions in future releases, or it might become unsupported.
type: "io.kestra.plugin.azure.runner.AzureBatchTaskRunner"
Task runner that executes a task inside a job in Azure Batch.
This task runner is container-based so the containerImage
property must be set.
To access the task's working directory, use the Pebble expression or the
WORKING_DIR
environment variable. Input files and namespace files will be available in this directory.
To generate output files you can either use the outputFiles
task's property and create a file with the same name in the task's working directory, or create any file in the output directory which can be accessed by the Pebble expression or the
OUTPUT_DIR
environment variables.
To use inputFiles
, outputFiles
or namespaceFiles
properties, make sure to set the blobStorage
property. The blob storage serves as an intermediary storage layer for the task runner. Input and namespace files will be uploaded to the cloud storage bucket before the task run. Similarly, the task runner will store outputFiles in this blob storage during the task run. In the end, the task runner will make those files available for download and preview from the UI by sending them to internal storage.
To make it easier to track where all files are stored, the task runner will generate a folder for each task run. You can access that folder using the Pebble expression or the
BUCKET_PATH
environment variable.
Note that when the Kestra Worker running this task is terminated, the batch job will still run until completion.
Examples
Execute a Shell command.
id: new-shell
namespace: myteam
tasks:
- id: shell
type: io.kestra.plugin.scripts.shell.Commands
taskRunner:
type: io.kestra.plugin.azure.runner.AzureBatchTaskRunner
account: "{{secrets.account}}"
accessKey: "{{secrets.accessKey}}"
endpoint: "{{secrets.endpoint}}"
poolId: "{{vars.poolId}}"
commands:
- echo "Hello World"
Pass input files to the task, execute a Shell command, then retrieve output files.
id: new-shell-with-file
namespace: myteam
inputs:
- id: file
type: FILE
tasks:
- id: shell
type: io.kestra.plugin.scripts.shell.Commands
inputFiles:
data.txt: "{{inputs.file}}"
outputFiles:
- out.txt
containerImage: centos
taskRunner:
type: io.kestra.plugin.azure.runner.AzureBatchTaskRunner
account: "{{secrets.account}}"
accessKey: "{{secrets.accessKey}}"
endpoint: "{{secrets.endpoint}}"
poolId: "{{vars.poolId}}"
blobStorage:
connectionString: "{{secrets.connectionString}}"
containerName: "{{vars.containerName}}"
commands:
- cp {{workingDir}}/data.txt {{workingDir}}/out.txt
Properties
accessKey
- Type: string
- Dynamic: ✔️
- Required: ✔️
The Batch access key.
account
- Type: string
- Dynamic: ✔️
- Required: ✔️
The Batch account name.
endpoint
- Type: string
- Dynamic: ✔️
- Required: ✔️
The blob service endpoint.
poolId
- Type: string
- Dynamic: ✔️
- Required: ✔️
Id of the pool on which to run the job.
blobStorage
- Type: BlobStorageForBatch
- Dynamic: ❓
- Required: ❌
registry
- Type: ContainerRegistry
- Dynamic: ❓
- Required: ❌
The private registry which contains the container image.
waitUntilCompletion
- Type: string
- Dynamic: ❓
- Required: ❌
- Default:
3600.000000000
- Format:
duration
The maximum duration to wait for the job completion. Azure Batch will automatically timeout the job upon reaching such duration and the task will be failed.
Definitions
io.kestra.core.models.tasks.retrys.Constant
Properties
interval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
type
- Type: string
- Dynamic: ❓
- Required: ✔️
- Default:
constant
behavior
- Type: string
- Dynamic: ❓
- Required: ❌
- Default:
RETRY_FAILED_TASK
- Possible Values:
RETRY_FAILED_TASK
CREATE_NEW_EXECUTION
maxAttempt
- Type: integer
- Dynamic: ❓
- Required: ❌
- Minimum:
>= 1
maxDuration
- Type: string
- Dynamic: ❓
- Required: ❌
- Format:
duration
warningOnRetry
- Type: boolean
- Dynamic: ❓
- Required: ❌
- Default:
false
io.kestra.core.models.tasks.retrys.Random
Properties
maxInterval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
minInterval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
type
- Type: string
- Dynamic: ❓
- Required: ✔️
- Default:
random
behavior
- Type: string
- Dynamic: ❓
- Required: ❌
- Default:
RETRY_FAILED_TASK
- Possible Values:
RETRY_FAILED_TASK
CREATE_NEW_EXECUTION
maxAttempt
- Type: integer
- Dynamic: ❓
- Required: ❌
- Minimum:
>= 1
maxDuration
- Type: string
- Dynamic: ❓
- Required: ❌
- Format:
duration
warningOnRetry
- Type: boolean
- Dynamic: ❓
- Required: ❌
- Default:
false
io.kestra.plugin.azure.batch.models.ContainerRegistry
Properties
identityReference
- Type: ComputeNodeIdentityReference
- Dynamic: ✔️
- Required: ❌
The reference to the user assigned identity to use to access the Azure Container Registry instead of username and password.
password
- Type: string
- Dynamic: ✔️
- Required: ❌
The password to log into the registry server.
registryServer
- Type: string
- Dynamic: ✔️
- Required: ❌
The registry server URL.
If omitted, the default is "docker.io".
userName
- Type: string
- Dynamic: ✔️
- Required: ❌
The user name to log into the registry server.
io.kestra.plugin.azure.storage.blob.models.BlobStorageForBatch
Properties
containerName
- Type: string
- Dynamic: ✔️
- Required: ✔️
The URL of the blob container the compute node should use.
Mandatory if you want to use
namespaceFiles
,inputFiles
oroutputFiles
properties.
endpoint
- Type: string
- Dynamic: ✔️
- Required: ✔️
The blob service endpoint.
id
- Type: string
- Dynamic: ❓
- Required: ✔️
- Validation regExp:
^[a-zA-Z0-9][a-zA-Z0-9_-]*
- Min length:
1
allowFailure
- Type: boolean
- Dynamic: ❓
- Required: ❌
- Default:
false
connectionString
- Type: string
- Dynamic: ✔️
- Required: ❌
Connection string of the Storage Account.
description
- Type: string
- Dynamic: ❓
- Required: ❌
disabled
- Type: boolean
- Dynamic: ❓
- Required: ❌
- Default:
false
logLevel
- Type: string
- Dynamic: ❓
- Required: ❌
- Possible Values:
ERROR
WARN
INFO
DEBUG
TRACE
retry
- Type:
- Dynamic: ❓
- Required: ❌
sasToken
- Type: string
- Dynamic: ✔️
- Required: ❌
The SAS token to use for authenticating requests.
This string should only be the query parameters (with or without a leading '?') and not a full URL.
sharedKeyAccountAccessKey
- Type: string
- Dynamic: ✔️
- Required: ❌
Shared Key access key for authenticating requests.
sharedKeyAccountName
- Type: string
- Dynamic: ✔️
- Required: ❌
Shared Key account name for authenticating requests.
timeout
- Type: string
- Dynamic: ❓
- Required: ❌
- Format:
duration
workerGroup
- Type: WorkerGroup
- Dynamic: ❓
- Required: ❌
io.kestra.plugin.azure.batch.models.ComputeNodeIdentityReference
Properties
resourceId
- Type: string
- Dynamic: ✔️
- Required: ❌
The ARM resource ID of the user assigned identity.
io.kestra.core.models.tasks.retrys.Exponential
Properties
interval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
maxInterval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
type
- Type: string
- Dynamic: ❓
- Required: ✔️
- Default:
exponential
behavior
- Type: string
- Dynamic: ❓
- Required: ❌
- Default:
RETRY_FAILED_TASK
- Possible Values:
RETRY_FAILED_TASK
CREATE_NEW_EXECUTION
delayFactor
- Type: number
- Dynamic: ❓
- Required: ❌
maxAttempt
- Type: integer
- Dynamic: ❓
- Required: ❌
- Minimum:
>= 1
maxDuration
- Type: string
- Dynamic: ❓
- Required: ❌
- Format:
duration
warningOnRetry
- Type: boolean
- Dynamic: ❓
- Required: ❌
- Default:
false
io.kestra.core.models.tasks.WorkerGroup
Properties
key
- Type: string
- Dynamic: ❓
- Required: ❌
- Validation regExp:
^[a-zA-Z0-9][a-zA-Z0-9_-]*