Create

type: "io.kestra.plugin.gcp.dataproc.clusters.Create"

Create clusters in Google Cloud Dataproc.

Examples

Creates a cluster in Google Cloud Dataproc.

id: gcp_dataproc_cluster_create
namespace: company.team

tasks:
  - id: cluster_create
    type: io.kestra.plugin.gcp.dataproc.clusters.Create
    clusterName: YOUR_CLUSTER_NAME
    region: YOUR_REGION
    zone: YOUR_ZONE
    masterMachineType: n1-standard-2
    workerMachineType: n1-standard-2
    workers: 2
    bucket: YOUR_BUCKET_NAME

Creates a cluster in Google Cloud Dataproc with specific disk size.

id: gcp_dataproc_cluster_create
namespace: company.team

tasks:
  - id: create_cluster_with_certain_disk_size
    type: io.kestra.plugin.gcp.dataproc.clusters.Create
    clusterName: YOUR_CLUSTER_NAME
    region: YOUR_REGION
    zone: YOUR_ZONE
    masterMachineType: n1-standard-2
    masterDiskSizeGB: 500
    workerMachineType: n1-standard-2
    workerDiskSizeGB: 200
    workers: 2
    bucket: YOUR_BUCKET_NAM

Properties

clusterName

• Type: string
• Dynamic: ✔️
• Required: ✔️
• Min length: 1

The cluster name.

region

• Type: string
• Dynamic: ✔️
• Required: ✔️
• Min length: 1

The region.

bucket

• Type: string
• Dynamic: ✔️
• Required: ❌

The GCS bucket name.

imageVersion

• Type: string
• Dynamic: ✔️
• Required: ❌

The Dataproc image URI.

The Compute Engine image resource used for cluster instances.

masterDiskSizeGB

• Type: integer
• Dynamic: ❌
• Required: ❌

The disk size in GB for each master node.

masterMachineType

• Type: string
• Dynamic: ✔️
• Required: ❌

The master machine type.

projectId

• Type: string
• Dynamic: ✔️
• Required: ❌

The GCP project ID.

scopes

• Type: array
• SubType: string
• Dynamic: ✔️
• Required: ❌
• Default: [https://www.googleapis.com/auth/cloud-platform]

The GCP scopes to be used.

serviceAccount

• Type: string
• Dynamic: ✔️
• Required: ❌

The GCP service account key.

workerDiskSizeGB

• Type: integer
• Dynamic: ❌
• Required: ❌

The disk size in GB for each worker node.

workerMachineType

• Type: string
• Dynamic: ✔️
• Required: ❌

The worker machine type.

workers

• Type: integer
• Dynamic: ❌
• Required: ❌

The number of workers.

zone

• Type: string
• Dynamic: ✔️
• Required: ❌

The zone.

Outputs

clusterName

• Type: string
• Required: ❌

The cluster name.

created

• Type: boolean
• Required: ❌
• Default: false

Whether cluster was created successfully.

Definitions