Azure SubmitCommandJob

Azure SubmitCommandJob
Certified
Submit a command job to an Azure Machine Learning compute target

Submits a single command job (e.g. a training script) to a compute cluster or instance and, by default, waits for it to reach a terminal state, exposing MLflow-backed metrics as task outputs. Killing the Kestra execution cancels the underlying Azure ML job. Defaults: wait=true, checkFrequency.interval=PT10S, checkFrequency.maxDuration=PT1H, cancelOnTimeout=true.

yaml
type: io.kestra.plugin.azure.ml.SubmitCommandJob

Examples

yaml
id: azure_ml_submit_command_job
namespace: company.team

tasks:
  - id: train
    type: io.kestra.plugin.azure.ml.SubmitCommandJob
    tenantId: "{{ secret('AZURE_TENANT_ID') }}"
    clientId: "{{ secret('AZURE_CLIENT_ID') }}"
    clientSecret: "{{ secret('AZURE_CLIENT_SECRET') }}"
    subscriptionId: "{{ secret('AZURE_SUBSCRIPTION_ID') }}"
    resourceGroupName: ml-rg
    workspaceName: ml-workspace
    computeName: cpu-cluster
    environmentId: "azureml:AzureML-sklearn-1.5:1"
    command: "python train.py --epochs 10"

  - id: check_accuracy
    type: io.kestra.plugin.core.execution.Fail
    condition: "{{ outputs.train.metrics['accuracy'] < 0.9 }}"

Properties

command *string

Command

Shell command executed on the compute target, e.g. python train.py --epochs 10

computeName *string

Compute target name

Name of an existing Azure Machine Learning compute cluster or compute instance

resourceGroupName *string

Resource group name

Resource group containing the Machine Learning workspace

subscriptionId *string

Subscription ID

Azure subscription GUID that owns the Machine Learning workspace

tenantId *string

Azure AD tenant ID (GUID)

workspaceName *string

Workspace name

Name of the Azure Machine Learning workspace

cancelOnTimeout booleanstring

Defaulttrue

Cancel the job on timeout

When wait=true and checkFrequency.maxDuration is exceeded, cancel the Azure ML job before failing the task; defaults to true

checkFrequency

Default

{
  "maxDuration": "PT1H",
  "interval": "PT10S"
}

Polling frequency

Interval and max duration used when wait=true

Definitions

io.kestra.plugin.azure.ml.SubmitCommandJob-CheckFrequency

intervalstring

DefaultPT10S

Polling interval

Delay between status checks; defaults to PT10S. Azure ML jobs run for minutes to hours, so a tighter interval only adds needless ARM API calls

maxDurationstring

DefaultPT1H

Max wait duration

Stop polling and fail after this duration; defaults to PT1H

clientId string

Client ID of the Azure AD application

Application (client) ID used for service principal authentication.

clientSecret string

Client secret for the Azure AD application

Secret value associated with the client ID; store in a Kestra secret.

displayName string

Display name

Human-readable run name shown in Azure ML Studio

environmentId string

Environment ID

Registered environment reference, e.g. azureml: AzureML-sklearn-1.5: 1; required by Azure Machine Learning to run the command

experimentName string

Experiment name

Groups this run under an experiment in Azure ML Studio

inputs object

Data inputs

Named folder inputs available to the command as ${{inputs.<name>}}; keys are input names, values are storage URIs such as azureml://datastores/<store>/paths/<path>.

instanceCount integerstring

Default1

Instance count

Number of compute nodes to use for this job; defaults to 1

name string

Job name

Unique job name within the workspace; a random UUID is generated when not set

outputs object

Data outputs

Named folder outputs the command writes to via ${{outputs.<name>}}; keys are output names, values are destination storage URIs.

pemCertificate string

PEM-encoded certificate content for client authentication

PEM text for certificate-based auth; alternative to clientSecret.

wait booleanstring

Defaulttrue

Wait for completion

If true (default), poll the job status until it reaches a terminal state and collect metrics

Outputs

jobName string

Job name

Name of the submitted Azure Machine Learning job

metrics object

SubTypenumber

Metrics

Metrics logged by the run, keyed by metric name. Always empty when wait=false, since the job has not necessarily finished logging anything yet.

Azure Machine Learning logs job metrics through MLflow, not through the ARM control-plane API used for everything else in this task. This value is fetched best-effort by reading the workspace's MLflow tracking URI and calling its REST API directly with the same Azure AD bearer token used to authenticate this task. If that call fails (e.g. the service principal lacks the required scope, or the endpoint is unreachable), a warning is logged and this field is an empty map — the task does not fail because of it.

outputs object

SubTypestring

Outputs

Named job outputs, keyed by output name, pointing to their storage URI

status string

Possible Values

NOT_STARTEDSTARTINGPROVISIONINGPREPARINGQUEUEDRUNNINGFINALIZINGCANCEL_REQUESTEDCOMPLETEDFAILEDCANCELEDNOT_RESPONDINGPAUSEDUNKNOWN

Job status

Terminal or last-observed job status

studioUrl string

Studio URL

Deep link to the run in Azure ML Studio