Google Cloud SparkSubmit

Google Cloud SparkSubmit

Certified

Submit a Spark batch to Dataproc

Runs a Spark batch using Dataproc Serverless or clusters. Provide main class and supporting JARs/files; args are passed to the driver.

yaml
type: io.kestra.plugin.gcp.dataproc.batches.SparkSubmit
yaml
id: gcp_dataproc_spark_submit
namespace: company.team
tasks:
  - id: spark_submit
    type: io.kestra.plugin.gcp.dataproc.batches.SparkSubmit
    jarFileUris:
      - 'gs://spark-jobs-kestra/spark-examples.jar'
    mainClass: org.apache.spark.examples.SparkPi
    args:
      - 1000
    name: test-spark
    region: europe-west3
Properties

Driver main class

Fully qualified class name; its JAR must be on the classpath or in jarFileUris

Batch name

Base name; a slugified suffix with execution id is appended and truncated to 63 chars

Dataproc region

Region endpoint used for the batch controller

SubTypestring

Archive URIs

HCFS URIs of archives extracted into each executor dir (.jar, .tar, .tar.gz, .tgz, .zip)

SubTypestring

Driver arguments

Arguments passed to the driver; avoid options that belong in batch properties (e.g., --conf)

Execution configuration

Network, service account, and KMS settings for the workload

Definitions
kmsKeystring

KMS key

Cloud KMS key for encryption

networkTagsarray
SubTypestring

Network tags

Applied to workload for network control

networkUristring

Network URI

VPC network URI for the workload

serviceAccountEmailstring

Service account

Email of the service account used to run the batch

subnetworkUristring

Subnetwork URI

Optional subnetwork URI for the workload

SubTypestring

File URIs

HCFS URIs copied into each executor working dir (gs://, hdfs://, or file://)

The GCP service account to impersonate

SubTypestring

Jar URIs

HCFS URIs of JARs added to driver and executor classpaths

Peripherals configuration

Optional Metastore and Spark History Server settings

Definitions
metastoreServicestring

Metastore service

Resource name of an existing Dataproc Metastore service (projects/{project}/locations/{region}/services/{id})

sparkHistoryServer

Spark History Server

Optional Dataproc cluster to serve Spark History for the workload

dataprocClusterstring

Spark History Server cluster

Dataproc cluster resource name (projects/{project}/regions/{region}/clusters/{name})

Reference (ref) of the pluginDefaults to apply to this task.

The GCP project ID

Runtime configuration

Container image, runtime version, and properties

Definitions
containerImagestring

Container image

Optional custom runtime image; defaults to Dataproc image when unset

propertiesobject

Runtime properties

Key/value pairs passed to the runtime

versionstring

Runtime version

Optional Dataproc runtime version string

SubTypestring
Default["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used

The GCP service account

Possible Values
STATE_UNSPECIFIEDPENDINGRUNNINGCANCELLINGCANCELLEDSUCCEEDEDFAILEDUNRECOGNIZED

Batch state

More Plugins in this Category

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks

PostgreSQL
PostgreSQL

Tasks that connect to PostgreSQL via JDBC to run queries and loads.

Data

6 tasks

25 blueprints