Azure ScaleCluster

Azure ScaleCluster

Certified

Update the autoscale settings of an Azure Machine Learning compute cluster

Sets the min/max node count (and optional idle scale-down delay) of an existing AmlCompute cluster. Only compute clusters support autoscaling; compute instances do not.

yaml
type: io.kestra.plugin.azure.ml.ScaleCluster
yaml
id: azure_ml_scale_cluster
namespace: company.team

tasks:
  - id: scale_up
    type: io.kestra.plugin.azure.ml.ScaleCluster
    tenantId: "{{ secret('AZURE_TENANT_ID') }}"
    clientId: "{{ secret('AZURE_CLIENT_ID') }}"
    clientSecret: "{{ secret('AZURE_CLIENT_SECRET') }}"
    subscriptionId: "{{ secret('AZURE_SUBSCRIPTION_ID') }}"
    resourceGroupName: ml-rg
    workspaceName: ml-workspace
    computeName: gpu-cluster
    minNodeCount: 1
    maxNodeCount: 4
Properties

Compute cluster name

Maximum node count

Maximum number of nodes the cluster can scale out to

Resource group name

Resource group containing the Machine Learning workspace

Subscription ID

Azure subscription GUID that owns the Machine Learning workspace

Azure AD tenant ID (GUID)

Workspace name

Name of the Azure Machine Learning workspace

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions
assetFailureBehaviorstring
Possible Values
IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

id*string
Min length1
typestring
outputs

The assets produced as outputs.

id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*string
Min length1

Custom asset type

descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150

Client ID of the Azure AD application

Application (client) ID used for service principal authentication.

Client secret for the Azure AD application

Secret value associated with the client ID; store in a Kestra secret.

Default0

Minimum node count

Minimum number of nodes kept available; 0 (default) scales the cluster down to no nodes when idle

Node idle time before scale down

How long a node stays idle before being deallocated; defaults to Azure's own setting when not provided

PEM-encoded certificate content for client authentication

PEM text for certificate-based auth; alternative to clientSecret.

Compute cluster name

Maximum node count

Minimum node count

More Plugins in this Category

Ashby
Ashby

Ashby plugin for Kestra

Data

1 tasks

1 blueprints

Dremio
Dremio

Tasks that connect to Dremio via JDBC to run queries and loads.

Data

3 tasks

3 blueprints