PLUGINSPLUGIN-DATABRICKSTASKSCLUSTERIO.KESTRA.PLUGIN.DATABRICKS.CLUSTER.CREATECLUSTER

​Create​Cluster

CreateCluster CreateCluster

yaml
type: "io.kestra.plugin.databricks.cluster.CreateCluster"

Create a Databricks cluster

Examples

Create a Databricks cluster with one worker

yaml
id: "create_cluster"
type: "io.kestra.plugin.databricks.cluster.CreateCluster"
id: createCluster
type: io.kestra.plugin.databricks.cluster.CreateCluster
authentication:
  token: <your-token>
host: <your-host>
clusterName: kestra-demo
nodeTypeId: n2-highmem-4
numWorkers: 1
sparkVersion: 13.0.x-scala2.12

Properties

clusterName

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The name of the cluster

sparkVersion

  • Type: string
  • Dynamic: ❌
  • Required: ✔️

The Spark version

accountId

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Databricks account identifier

authentication

Databricks authentication configuration

This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.

autoTerminationMinutes

  • Type: integer
  • Dynamic: ❌
  • Required: ❌

If set, the cluster will be terminated automatically after this time period

configFile

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one

host

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Databricks host

maxWorkers

  • Type: integer
  • Dynamic: ❌
  • Required: ❌

The maximum number of workers

Use this property along with minWorkers to use autoscaling. Otherwise, set a fixed number of workers using numWorkers.

minWorkers

  • Type: integer
  • Dynamic: ❌
  • Required: ❌

The minimum number of workers

Use this property along with maxWorkers for autoscaling. Otherwise, set a fixed number of workers using numWorkers.

nodeTypeId

  • Type: string
  • Dynamic: ❌
  • Required: ❌

The type of node, the value depends on the cloud provider

numWorkers

  • Type: integer
  • Dynamic: ❌
  • Required: ❌

The fixed number of workers

You must set this property unless you use the minWorkers and maxWorkers properties for autoscaling.

Outputs

clusterId

  • Type: string

The cluster identifier

clusterState

  • Type: string
  • Possible Values:
    • ERROR
    • PENDING
    • RESIZING
    • RESTARTING
    • RUNNING
    • TERMINATED
    • TERMINATING
    • UNKNOWN

The cluster state

clusterURI

  • Type: string

The cluster URI on the Databricks console

Definitions

AuthenticationConfig

authType

  • Type: string
  • Dynamic: ❌
  • Required: ❌

azureClientId

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

azureClientSecret

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

azureTenantId

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

clientId

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

clientSecret

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

googleCredentials

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

googleServiceAccount

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

password

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

token

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

username

  • Type: string
  • Dynamic: ✔️
  • Required: ❌