ESC
Ask Kestra AI
ESC
Kestra AI

Hi! I'm your Kestra AI assistant.
Ask me anything about workflows.

EXAMPLE QUESTIONS

How to run dbt?
How to configure my internal storage?
How to deploy Kestra inside Kubernetes?

Azure Create

Azure Create

Certified

Run tasks in an Azure Batch job

Creates or resumes a job on a target pool, uploads task definitions, waits for completion, and optionally deletes the job afterwards. Defaults: resume=true, delete=true, completionCheckInterval=PT1S; set maxDuration to cap wait time.

yaml
type: io.kestra.plugin.azure.batch.job.Create
yaml
id: azure_batch_job_create
namespace: company.team

tasks:
  - id: create
    type: io.kestra.plugin.azure.batch.job.Create
    endpoint: https://my.francecentral.batch.azure.com
    account: "{{ secret('AZURE_ACCOUNT') }}"
    accessKey: "{{ secret('AZURE_ACCESS_KEY') }}"
    poolId: "{{ secret('AZURE_POOL_ID') }}"
    job:
      id: myjob
    tasks:
      - id: env
        commands:
          - 'echo t1=$ENV_STRING'
        environments:
          ENV_STRING: "{{ inputs.first }}"

      - id: echo
        commands:
          - 'echo t2={{ inputs.second }} 1>&2'

      - id: for
        commands:
          -  'for i in $(seq 10); do echo t3=$i; done'

      - id: vars
        commands:
          - echo '::{"outputs":{"extract":"'$(cat files/in/in.txt)'"}::'
        resourceFiles:
          - httpUrl: https://unittestkt.blob.core.windows.net/tasks/***?sv=***&se=***&sr=***&sp=***&sig=***
            filePath: files/in/in.txt

      - id: output
        commands:
          - 'mkdir -p outs/child/sub'
          - 'echo 1 > outs/1.txt'
          - 'echo 2 > outs/child/2.txt'
          - 'echo 3 > outs/child/sub/3.txt'
        outputFiles:
          - outs/1.txt
        outputDirs:
          - outs/child

Use a container to start the task, the pool must use a microsoft-azure-batch publisher.

yaml
id: azure_batch_job_create
namespace: company.team

tasks:
  - id: create
    type: io.kestra.plugin.azure.batch.job.Create
    endpoint: https://***.francecentral.batch.azure.com
    account: "{{ secret('AZURE_ACCOUNT') }}"
    accessKey: "{{ secret('AZURE_ACCESS_KEY') }}"
    poolId: <pool-id>
    job:
      id: <job-name>
    tasks:
      - id: echo
        commands:
          - 'python --version'
        containerSettings:
          imageName: python
Properties

Service endpoint URL

Base HTTPS endpoint of the target service.

Job definition

Job metadata and scheduling limits passed to Azure

Definitions
id*string
Max length64

Job ID

Unique within the Batch account (<=64 chars, alphanumeric, hyphen, underscore); case-insensitive uniqueness

displayNamestring
Max length1024

Display name

Optional friendly name (up to 1024 Unicode chars); not required to be unique

labelsobject

Job labels

Key/value metadata applied to the job

maxParallelTasksintegerstring

Max parallel tasks

Set to -1 for unlimited (default) or a positive integer; controls concurrent task scheduling

priorityintegerstring

Job priority

Integer from -1000 to 1000; defaults to 0

Pool ID

Existing Batch pool where the job runs; must be active

Tasks to run

Ordered list of task definitions executed within the job

Definitions
commands*array
SubTypestring

Commands

Command lines executed inside the interpreter; rendered with expressions then joined and passed to interpreter/args

id*string
Max length64

Task ID

Unique within the job (<=64 chars, alphanumeric, hyphen, underscore); case-insensitive; generated if omitted

constraints

Execution constraints

maxTaskRetryCountintegerstring

Max retry count

Number of retries after non-zero exit: 0 disables retries; -1 retries indefinitely; default is Azure Batch default

maxWallClockTimestring

Max wall clock time

Duration limit from task start; Batch terminates the task if it exceeds this. Null means no limit.

retentionTimestring

Retention time

Minimum time to keep the task working directory after completion; default is 7 days

containerSettings

Container settings

Container image and run options; required if the pool is container-enabled, omit otherwise

imageName*string

Container image

Full image reference (e.g. repo/name: tag); defaults to : latest if no tag provided

containerRunOptionsstring

Container run options

Extra arguments passed to docker create alongside Batch-managed options

registry

Container registry

Registry credentials; omit when provided at pool creation

identityReference

User-assigned identity for ACR

Use managed identity instead of username/password when supported by the registry

passwordstring

Registry password

registryServerstring

Registry server

Container registry hostname; defaults to docker.io

userNamestring

Registry username

workingDirectorystring
Possible Values
TASK_WORKING_DIRECTORYCONTAINER_IMAGE_DEFAULT

Working directory scope

Where the container starts; defaults to TASK_WORKING_DIRECTORY. Options: TASK_WORKING_DIRECTORY or CONTAINER_IMAGE_DEFAULT.

displayNamestring
Max length1024

Display name

Optional friendly name (<=1024 Unicode chars); not required to be unique

environmentsobject
SubTypestring

Environment variables

Key/value environment map rendered before execution

interpreterstring
Default/bin/sh

Command interpreter

Shell used to run the task; defaults to /bin/sh

interpreterArgsarray
SubTypestring
Default[ "-c" ]

Interpreter arguments

Arguments prepended to the command; defaults to ['-c']

outputDirsarray
SubTypestring

Output directories

Directories to sync to internal storage; use {{ outputDirs.key }} to write files under that directory

outputFilesarray
SubTypestring

Output files

Names of files to upload from the working directory to internal storage; use {{ outputFiles.key }} to write paths

requiredSlotsintegerstring

Required scheduling slots

Number of slots needed on a node; default 1 and must remain 1 for multi-instance tasks

resourceFilesarray

Resource files

Files staged to the node before execution; subject to Azure Batch request size limits

autoStorageContainerNamestring

Auto storage container name

Mutually exclusive with storageContainerUrl and httpUrl; one of the three is required

blobPrefixstring

Blob prefix filter

Filters blobs when using autoStorageContainerName or storageContainerUrl; downloads only names starting with this prefix

fileModestring

File mode (octal)

Linux-only permission bits; defaults to 0770 on Linux; ignored on Windows nodes

filePathstring

Target path on node

Destination relative to task working dir; required for httpUrl (includes filename); optional dir for container downloads

httpUrlstring

HTTP/HTTPS file URL

Direct file URL; mutually exclusive with storageContainerUrl and autoStorageContainerName; must be readable from compute nodes

identityReference

User-assigned identity reference

Identity used to access storage when using storageContainerUrl or httpUrl

resourceIdstring

User-assigned identity resource ID

ARM resource ID of the managed identity used for storage access

storageContainerUrlstring

Blob container URL

SAS/identity/public URL to a container; mutually exclusive with autoStorageContainerName and httpUrl; must allow read+list from compute nodes

uploadFilesarray

Upload files

Files Azure Batch uploads from the node after execution; for multi-instance tasks only the primary node uploads

destination*

Upload destination

container*

Azure Blob destination

filePatternstring

File pattern to upload

Supports absolute or task-relative paths with wildcards (*, **, ?, [set]); env vars expanded before matching; dotfiles require explicit match

uploadOptions
Default{ "uploadCondition": "taskcompletion" }

Upload options

Controls when the upload runs and other transfer settings

uploadConditionstring
Defaulttaskcompletion
Possible Values
TASK_SUCCESSTASK_FAILURETASK_COMPLETION

Upload condition

When to upload files (e.g., TASK_COMPLETION, TASK_SUCCESS); defaults to TASK_COMPLETION

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions
assetFailureBehaviorstring
Possible Values
IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

id*string
Min length1
typestring
outputs

The assets produced as outputs.

id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*string
Min length1

Custom asset type

descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
DefaultPT1S

Completion check interval

Poll interval for job status while waiting; defaults to PT1S

Defaulttrue

Delete job on completion

If true (default), the job is deleted after tasks finish; keep false to inspect failures but retries may reuse the old job

Maximum wait duration

Timeout while waiting for all tasks to finish; null means wait indefinitely

Defaulttrue

Resume existing job if found

When true (default), reuses a matching in-progress job instead of creating a new one

Defaultfalse

Whether to synchronize working directory from remote runner back to local one after run.

SubTypestring

Output files stored in internal storage

Captured outputs from task logs

The CPU kernel time.

The CPU user time.

The number of gibibytes read from I/O by the task.

The number of read I/O operations performed by the task.

The number of gibibytes written to I/O by the task.

The number of write I/O operations performed by the task.

The wall clock time.

More Plugins in this Category

Scrapy CLI
Scrapy CLI

Tasks that run Scrapy spiders and CLI commands to scrape data from websites.

Data

1 tasks

Supabase
Supabase

Tasks that call Supabase APIs for data and storage operations.

Data

5 tasks