Databricks CreateCluster
CertifiedCreate a Databricks cluster
Databricks CreateCluster
Create a Databricks cluster
Provisions a new Databricks cluster via the Compute API. Set a fixed worker count with numWorkers or enable autoscaling with minWorkers/maxWorkers; auto termination is optional.
type: io.kestra.plugin.databricks.cluster.CreateClusterExamples
Create a Databricks cluster with one worker.
id: databricks_create_cluster
namespace: company.team
tasks:
- id: create_cluster
type: io.kestra.plugin.databricks.cluster.CreateCluster
authentication:
token: "{{ secret('DATABRICKS_TOKEN') }}"
host: <your-host>
clusterName: kestra-demo
nodeTypeId: n2-highmem-4
numWorkers: 1
sparkVersion: 13.0.x-scala2.12
Properties
clusterName *string
Cluster name
sparkVersion *string
Spark version
Runtime identifier, e.g. 13.0.x-scala2.12
accountId string
Databricks account identifier
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150authentication
Databricks authentication configuration
This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.
io.kestra.plugin.databricks.AbstractTask-AuthenticationConfig
Authentication type
Azure client ID
Azure client secret
Azure tenant ID
Client ID
Client secret
Google credentials JSON
Google service account email
Password
Databricks personal access token
Username
autoTerminationMinutes integerstring
Auto-termination minutes
Idle timeout; cluster is terminated after this duration if set
configFile string
Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one
host string
Databricks host
maxWorkers integerstring
Maximum workers
Use with minWorkers to enable autoscaling; ignored when numWorkers is set
minWorkers integerstring
Minimum workers
Use with maxWorkers to enable autoscaling; ignored when numWorkers is set
nodeTypeId string
Node type
Instance type; values depend on the workspace cloud provider
numWorkers integerstring
Fixed workers
Required unless autoscaling is configured; sets numWorkers on the cluster
Outputs
clusterId string
Cluster identifier
clusterState string
ERRORPENDINGRESIZINGRESTARTINGRUNNINGTERMINATEDTERMINATINGUNKNOWNCluster state
clusterURI string
uriCluster console URI
