Google Cloud Create
CertifiedCreate a Dataproc cluster
Google Cloud Create
Create a Dataproc cluster
Creates a Dataproc cluster with optional zone, image, machine shapes, disk sizes, worker count, and staging bucket. Uses project credentials; endpoint derived from region.
type: io.kestra.plugin.gcp.dataproc.clusters.CreateExamples
Creates a cluster in Google Cloud Dataproc
id: gcp_dataproc_cluster_create
namespace: company.team
tasks:
- id: cluster_create
type: io.kestra.plugin.gcp.dataproc.clusters.Create
clusterName: YOUR_CLUSTER_NAME
region: YOUR_REGION
zone: YOUR_ZONE
masterMachineType: n1-standard-2
workerMachineType: n1-standard-2
workers: 2
bucket: YOUR_BUCKET_NAME
Creates a cluster in Google Cloud Dataproc with specific disk size
id: gcp_dataproc_cluster_create
namespace: company.team
tasks:
- id: create_cluster_with_certain_disk_size
type: io.kestra.plugin.gcp.dataproc.clusters.Create
clusterName: YOUR_CLUSTER_NAME
region: YOUR_REGION
zone: YOUR_ZONE
masterMachineType: n1-standard-2
masterDiskSizeGB: 500
workerMachineType: n1-standard-2
workerDiskSizeGB: 200
workers: 2
bucket: YOUR_BUCKET_NAME
Properties
clusterName *string
1Cluster name
region *string
1Region
Dataproc region endpoint (e.g., europe-west3)
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150bucket string
Bucket
Optional GCS bucket for cluster staging
imageVersion string
Dataproc image version
Compute Engine image for instances (e.g., 2.1-debian12)
impersonatedServiceAccount string
The GCP service account to impersonate
masterDiskSizeGB integerstring
Master disk size (GB)
masterMachineType string
Master machine type
projectId string
The GCP project ID
scopes array
["https://www.googleapis.com/auth/cloud-platform"]The GCP scopes to be used
serviceAccount string
The GCP service account
workerDiskSizeGB integerstring
Worker disk size (GB)
workerMachineType string
Worker machine type
workers integerstring
Number of workers
zone string
Zone
Optional compute zone URI
Outputs
clusterName string
The cluster name
created boolean
falseWhether cluster was created successfully
