CreateCluster
Create a Databricks cluster.
Create a Databricks cluster.
Create a Databricks cluster.
type: "io.kestra.plugin.databricks.cluster.CreateCluster"Examples
Create a Databricks cluster with one worker.
id: databricks_create_cluster
namespace: company.team
tasks:
- id: create_cluster
type: io.kestra.plugin.databricks.cluster.CreateCluster
authentication:
token: "{{ secret('DATABRICKS_TOKEN') }}"
host: <your-host>
clusterName: kestra-demo
nodeTypeId: n2-highmem-4
numWorkers: 1
sparkVersion: 13.0.x-scala2.12
Properties
clusterName*Requiredstring
The name of the cluster.
sparkVersion*Requiredstring
The Spark version.
accountIdstring
Databricks account identifier.
authentication
Databricks authentication configuration.
This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.
io.kestra.plugin.databricks.AbstractTask-AuthenticationConfig
autoTerminationMinutesintegerstring
If set, the cluster will be terminated automatically after this time period.
configFilestring
Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one.
hoststring
Databricks host.
maxWorkersintegerstring
The maximum number of workers.
Use this property along with minWorkers to use autoscaling. Otherwise, set a fixed number of workers using numWorkers.
minWorkersintegerstring
The minimum number of workers.
Use this property along with maxWorkers for autoscaling. Otherwise, set a fixed number of workers using numWorkers.
nodeTypeIdstring
The type of node, the value depends on the cloud provider.
numWorkersintegerstring
The fixed number of workers.
You must set this property unless you use the minWorkers and maxWorkers properties for autoscaling.
Outputs
clusterIdstring
The cluster identifier.
clusterStatestring
ERRORPENDINGRESIZINGRESTARTINGRUNNINGTERMINATEDTERMINATINGUNKNOWNThe cluster state.
clusterURIstring
uriThe cluster URI on the Databricks console.
