Azure SubmitPipelineJob

Azure SubmitPipelineJob
Certified
Submit a multi-step pipeline job to Azure Machine Learning

Submits a pipeline job made of several child jobs (e.g. a data-preparation step followed by a training step) and, by default, waits for it to reach a terminal state. jobs is the raw pipeline job graph as accepted by the Azure Machine Learning REST API: a map of step name to step definition (type, computeId, command, environmentId, inputs, outputs, ...). Each step's computeId must be the compute's full ARM resource ID (/subscriptions/.../resourceGroups/.../providers/Microsoft.MachineLearningServices/workspaces/.../computes/<name>) — a bare compute name is rejected by the API. A type: command step that has no componentId gets a minimal Azure Machine Learning component automatically registered from its command/environmentId (named <job name>-<step name>, version 1), since Azure's pipeline job API only accepts steps that reference a component, unlike a standalone command job — set componentId explicitly on a step to reference an existing component instead and skip auto-registration for it. Killing the Kestra execution cancels the whole pipeline job. Defaults: wait=true, checkFrequency.interval=PT10S, checkFrequency.maxDuration=PT1H, cancelOnTimeout=true.

yaml
type: io.kestra.plugin.azure.ml.SubmitPipelineJob

Examples

yaml
id: azure_ml_submit_pipeline_job
namespace: company.team

tasks:
  - id: run_pipeline
    type: io.kestra.plugin.azure.ml.SubmitPipelineJob
    tenantId: "{{ secret('AZURE_TENANT_ID') }}"
    clientId: "{{ secret('AZURE_CLIENT_ID') }}"
    clientSecret: "{{ secret('AZURE_CLIENT_SECRET') }}"
    subscriptionId: "{{ secret('AZURE_SUBSCRIPTION_ID') }}"
    resourceGroupName: ml-rg
    workspaceName: ml-workspace
    jobs:
      prepare:
        type: command
        computeId: "/subscriptions/{{ secret('AZURE_SUBSCRIPTION_ID') }}/resourceGroups/ml-rg/providers/Microsoft.MachineLearningServices/workspaces/ml-workspace/computes/cpu-cluster"
        command: "python prepare.py"
        environmentId: "azureml:AzureML-sklearn-1.5:1"
      train:
        type: command
        computeId: "/subscriptions/{{ secret('AZURE_SUBSCRIPTION_ID') }}/resourceGroups/ml-rg/providers/Microsoft.MachineLearningServices/workspaces/ml-workspace/computes/cpu-cluster"
        command: "python train.py"
        environmentId: "azureml:AzureML-sklearn-1.5:1"

Properties

jobs *object

Pipeline steps

Raw pipeline job graph, as a map of step name to step definition, following the Azure Machine Learning REST API jobs schema for pipeline jobs.

resourceGroupName *string

Resource group name

Resource group containing the Machine Learning workspace

subscriptionId *string

Subscription ID

Azure subscription GUID that owns the Machine Learning workspace

tenantId *string

Azure AD tenant ID (GUID)

workspaceName *string

Workspace name

Name of the Azure Machine Learning workspace

cancelOnTimeout booleanstring

Defaulttrue

Cancel the pipeline on timeout

When wait=true and checkFrequency.maxDuration is exceeded, cancel the Azure ML pipeline job before failing the task; defaults to true

checkFrequency

Default

{
  "maxDuration": "PT1H",
  "interval": "PT10S"
}

Polling frequency

Interval and max duration used when wait=true

Definitions

io.kestra.plugin.azure.ml.SubmitCommandJob-CheckFrequency

intervalstring

DefaultPT10S

Polling interval

Delay between status checks; defaults to PT10S. Azure ML jobs run for minutes to hours, so a tighter interval only adds needless ARM API calls

maxDurationstring

DefaultPT1H

Max wait duration

Stop polling and fail after this duration; defaults to PT1H

clientId string

Client ID of the Azure AD application

Application (client) ID used for service principal authentication.

clientSecret string

Client secret for the Azure AD application

Secret value associated with the client ID; store in a Kestra secret.

displayName string

Display name

Human-readable run name shown in Azure ML Studio

experimentName string

Experiment name

Groups this run under an experiment in Azure ML Studio

name string

Job name

Unique job name within the workspace; a random UUID is generated when not set

pemCertificate string

PEM-encoded certificate content for client authentication

PEM text for certificate-based auth; alternative to clientSecret.

wait booleanstring

Defaulttrue

Wait for completion

If true (default), poll the pipeline status until it reaches a terminal state

Outputs

jobName string

Job name

Name of the submitted Azure Machine Learning pipeline job

metrics object

SubTypenumber

Metrics

Metrics logged by the run, keyed by metric name. Always empty when wait=false, since the pipeline has not necessarily finished logging anything yet.

Azure Machine Learning logs job metrics through MLflow, not through the ARM control-plane API used for everything else in this task. This value is fetched best-effort by reading the workspace's MLflow tracking URI and calling its REST API directly with the same Azure AD bearer token used to authenticate this task. If that call fails (e.g. the service principal lacks the required scope, or the endpoint is unreachable), a warning is logged and this field is an empty map — the task does not fail because of it.

outputs object

SubTypestring

Outputs

Named pipeline outputs, keyed by output name, pointing to their storage URI

status string

Possible Values

NOT_STARTEDSTARTINGPROVISIONINGPREPARINGQUEUEDRUNNINGFINALIZINGCANCEL_REQUESTEDCOMPLETEDFAILEDCANCELEDNOT_RESPONDINGPAUSEDUNKNOWN

Job status

Terminal or last-observed job status

studioUrl string

Studio URL

Deep link to the run in Azure ML Studio