CreateCluster
CreateCluster
type: "io.kestra.plugin.databricks.cluster.CreateCluster"
Create a Databricks cluster
Examples
Create a Databricks cluster with one worker
id: "create_cluster"
type: "io.kestra.plugin.databricks.cluster.CreateCluster"
id: createCluster
type: io.kestra.plugin.databricks.cluster.CreateCluster
authentication:
token: <your-token>
host: <your-host>
clusterName: kestra-demo
nodeTypeId: n2-highmem-4
numWorkers: 1
sparkVersion: 13.0.x-scala2.12
Properties
clusterName
- Type: string
- Dynamic: ✔️
- Required: ✔️
The name of the cluster
sparkVersion
- Type: string
- Dynamic: ❌
- Required: ✔️
The Spark version
accountId
- Type: string
- Dynamic: ✔️
- Required: ❌
Databricks account identifier
authentication
- Type: AuthenticationConfig
- Dynamic: ❌
- Required: ❌
Databricks authentication configuration
This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.
autoTerminationMinutes
- Type: integer
- Dynamic: ❌
- Required: ❌
If set, the cluster will be terminated automatically after this time period
configFile
- Type: string
- Dynamic: ✔️
- Required: ❌
Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one
host
- Type: string
- Dynamic: ✔️
- Required: ❌
Databricks host
maxWorkers
- Type: integer
- Dynamic: ❌
- Required: ❌
The maximum number of workers
Use this property along with minWorkers
to use autoscaling. Otherwise, set a fixed number of workers using numWorkers
.
minWorkers
- Type: integer
- Dynamic: ❌
- Required: ❌
The minimum number of workers
Use this property along with maxWorkers
for autoscaling. Otherwise, set a fixed number of workers using numWorkers
.
nodeTypeId
- Type: string
- Dynamic: ❌
- Required: ❌
The type of node, the value depends on the cloud provider
numWorkers
- Type: integer
- Dynamic: ❌
- Required: ❌
The fixed number of workers
You must set this property unless you use the minWorkers
and maxWorkers
properties for autoscaling.
Outputs
clusterId
- Type: string
The cluster identifier
clusterState
- Type: string
- Possible Values:
ERROR
PENDING
RESIZING
RESTARTING
RUNNING
TERMINATED
TERMINATING
UNKNOWN
The cluster state
clusterURI
- Type: string
The cluster URI on the Databricks console
Definitions
AuthenticationConfig
authType
- Type: string
- Dynamic: ❌
- Required: ❌
azureClientId
- Type: string
- Dynamic: ✔️
- Required: ❌
azureClientSecret
- Type: string
- Dynamic: ✔️
- Required: ❌
azureTenantId
- Type: string
- Dynamic: ✔️
- Required: ❌
clientId
- Type: string
- Dynamic: ✔️
- Required: ❌
clientSecret
- Type: string
- Dynamic: ✔️
- Required: ❌
googleCredentials
- Type: string
- Dynamic: ✔️
- Required: ❌
googleServiceAccount
- Type: string
- Dynamic: ✔️
- Required: ❌
password
- Type: string
- Dynamic: ✔️
- Required: ❌
token
- Type: string
- Dynamic: ✔️
- Required: ❌
username
- Type: string
- Dynamic: ✔️
- Required: ❌