Google Cloud PySparkSubmit

Google Cloud PySparkSubmit

Certified

Submit a PySpark batch to Dataproc

Runs a PySpark batch from a main Python file; supports extra JARs, files, archives, and args.

yaml
type: io.kestra.plugin.gcp.dataproc.batches.PySparkSubmit
yaml
id: gcp_dataproc_py_spark_submit
namespace: company.team
tasks:
  - id: py_spark_submit
    type: io.kestra.plugin.gcp.dataproc.batches.PySparkSubmit
    mainPythonFileUri: 'gs://spark-jobs-kestra/pi.py'
    name: test-pyspark
    region: europe-west3
Properties

Main Python file URI

HCFS URI to the driver .py file (gs://, hdfs://, or file://)

Batch name

Base name; a slugified suffix with execution id is appended and truncated to 63 chars

Dataproc region

Region endpoint used for the batch controller

SubTypestring

Archive URIs

HCFS URIs of archives extracted into each executor dir (.jar, .tar, .tar.gz, .tgz, .zip)

SubTypestring

Driver arguments

Arguments passed to the driver; avoid options that belong in batch properties (e.g., --conf)

Execution configuration

Network, service account, and KMS settings for the workload

Definitions
kmsKeystring

KMS key

Cloud KMS key for encryption

networkTagsarray
SubTypestring

Network tags

Applied to workload for network control

networkUristring

Network URI

VPC network URI for the workload

serviceAccountEmailstring

Service account

Email of the service account used to run the batch

subnetworkUristring

Subnetwork URI

Optional subnetwork URI for the workload

SubTypestring

File URIs

HCFS URIs copied into each executor working dir (gs://, hdfs://, or file://)

The GCP service account to impersonate

SubTypestring

Jar URIs

HCFS URIs of JARs added to driver and executor classpaths

Peripherals configuration

Optional Metastore and Spark History Server settings

Definitions
metastoreServicestring

Metastore service

Resource name of an existing Dataproc Metastore service (projects/{project}/locations/{region}/services/{id})

sparkHistoryServer

Spark History Server

Optional Dataproc cluster to serve Spark History for the workload

dataprocClusterstring

Spark History Server cluster

Dataproc cluster resource name (projects/{project}/regions/{region}/clusters/{name})

Reference (ref) of the pluginDefaults to apply to this task.

The GCP project ID

Runtime configuration

Container image, runtime version, and properties

Definitions
containerImagestring

Container image

Optional custom runtime image; defaults to Dataproc image when unset

propertiesobject

Runtime properties

Key/value pairs passed to the runtime

versionstring

Runtime version

Optional Dataproc runtime version string

SubTypestring
Default["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used

The GCP service account

Possible Values
STATE_UNSPECIFIEDPENDINGRUNNINGCANCELLINGCANCELLEDSUCCEEDEDFAILEDUNRECOGNIZED

Batch state

More Plugins in this Category

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks

OPA (Open Policy Agent)
OPA (Open Policy Agent)
Enterprise

Tasks to manage Open Policy Agent policies and decisions, including upload, evaluation, and partial evaluation.

Infrastructure

5 tasks

3 blueprints