PySparkSubmit

yaml
type: "io.kestra.plugin.gcp.dataproc.batches.PySparkSubmit"

Examples

yaml
id: gcp_dataproc_py_spark_submit
namespace: company.team
tasks:
  - id: py_spark_submit
    type: io.kestra.plugin.gcp.dataproc.batches.PySparkSubmit
    mainPythonFileUri: 'gs://spark-jobs-kestra/pi.py'
    name: test-pyspark
    region: europe-west3

Properties

mainPythonFileUri *string

name *string

region *string

archiveUris array

SubTypestring

args array

SubTypestring

execution

Definitions

io.kestra.plugin.gcp.dataproc.batches.AbstractBatch-ExecutionConfiguration

kmsKeystring

networkTagsarray

SubTypestring

networkUristring

serviceAccountEmailstring

subnetworkUristring

fileUris array

SubTypestring

impersonatedServiceAccount string

jarFileUris array

SubTypestring

peripherals

Definitions

io.kestra.plugin.gcp.dataproc.batches.AbstractBatch-PeripheralsConfiguration

metastoreServicestring

sparkHistoryServer

io.kestra.plugin.gcp.dataproc.batches.AbstractBatch-SparkHistoryServerConfiguration

dataprocClusterstring

projectId string

runtime

Definitions

io.kestra.plugin.gcp.dataproc.batches.AbstractBatch-RuntimeConfiguration

containerImagestring

propertiesobject

SubTypestring

versionstring

scopes array

SubTypestring

Default["https://www.googleapis.com/auth/cloud-platform"]

serviceAccount string

Outputs

state string

Possible Values

STATE_UNSPECIFIEDPENDINGRUNNINGCANCELLINGCANCELLEDSUCCEEDEDFAILEDUNRECOGNIZED

Core Plugins and tasks

Tasks that provide Kestra's built-in orchestration, I/O, and observability capabilities.

CoreStorageScript

PostgreSQL

Tasks that connect to PostgreSQL via JDBC to run queries and loads.

Database

Couchbase

Tasks that execute Couchbase N1QL queries and trigger Kestra flows from query results.

Database

PySparkSubmit

More Plugins in this Category

Core Plugins and tasks

PostgreSQL

Couchbase

2.0.0