Databricks CreateCluster

Databricks CreateCluster

Certified

Create a Databricks cluster

Provisions a new Databricks cluster via the Compute API. Set a fixed worker count with numWorkers or enable autoscaling with minWorkers/maxWorkers; auto termination is optional.

yaml
type: io.kestra.plugin.databricks.cluster.CreateCluster

Create a Databricks cluster with one worker.

yaml
id: databricks_create_cluster
namespace: company.team

tasks:
  - id: create_cluster
    type: io.kestra.plugin.databricks.cluster.CreateCluster
    authentication:
      token: "{{ secret('DATABRICKS_TOKEN') }}"
    host: <your-host>
    clusterName: kestra-demo
    nodeTypeId: n2-highmem-4
    numWorkers: 1
    sparkVersion: 13.0.x-scala2.12
Properties

Cluster name

Spark version

Runtime identifier, e.g. 13.0.x-scala2.12

Databricks account identifier

Databricks authentication configuration

This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.

Definitions
authTypestring

Authentication type

azureClientIdstring

Azure client ID

azureClientSecretstring

Azure client secret

azureTenantIdstring

Azure tenant ID

clientIdstring

Client ID

clientSecretstring

Client secret

googleCredentialsstring

Google credentials JSON

googleServiceAccountstring

Google service account email

passwordstring

Password

tokenstring

Databricks personal access token

usernamestring

Username

Auto-termination minutes

Idle timeout; cluster is terminated after this duration if set

Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one

Databricks host

Maximum workers

Use with minWorkers to enable autoscaling; ignored when numWorkers is set

Minimum workers

Use with maxWorkers to enable autoscaling; ignored when numWorkers is set

Node type

Instance type; values depend on the workspace cloud provider

Fixed workers

Required unless autoscaling is configured; sets numWorkers on the cluster

Reference (ref) of the pluginDefaults to apply to this task.

Cluster identifier

Possible Values
ERRORPENDINGRESIZINGRESTARTINGRUNNINGTERMINATEDTERMINATINGUNKNOWN

Cluster state

Formaturi

Cluster console URI

More Plugins in this Category

Clickhouse
Clickhouse

Tasks that connect to ClickHouse via JDBC to run queries and loads.

Data

5 tasks

2 blueprints

Weaviate
Weaviate

Tasks that manage and query vectors in Weaviate.

DataAI

4 tasks

2 blueprints