SparkSubmit

yaml
type: "io.kestra.plugin.gcp.dataproc.batches.SparkSubmit"

Examples

yaml
id: gcp_dataproc_spark_submit
namespace: company.team
tasks:
  - id: spark_submit
    type: io.kestra.plugin.gcp.dataproc.batches.SparkSubmit
    jarFileUris:
      - 'gs://spark-jobs-kestra/spark-examples.jar'
    mainClass: org.apache.spark.examples.SparkPi
    args:
      - 1000
    name: test-spark
    region: europe-west3

Properties

mainClass *string

name *string

region *string

archiveUris array

SubTypestring

args array

SubTypestring

execution

Definitions

io.kestra.plugin.gcp.dataproc.batches.AbstractBatch-ExecutionConfiguration

kmsKeystring

networkTagsarray

SubTypestring

networkUristring

serviceAccountEmailstring

subnetworkUristring

fileUris array

SubTypestring

impersonatedServiceAccount string

jarFileUris array

SubTypestring

peripherals

Definitions

io.kestra.plugin.gcp.dataproc.batches.AbstractBatch-PeripheralsConfiguration

metastoreServicestring

sparkHistoryServer

io.kestra.plugin.gcp.dataproc.batches.AbstractBatch-SparkHistoryServerConfiguration

dataprocClusterstring

projectId string

runtime

Definitions

io.kestra.plugin.gcp.dataproc.batches.AbstractBatch-RuntimeConfiguration

containerImagestring

propertiesobject

SubTypestring

versionstring

scopes array

SubTypestring

Default["https://www.googleapis.com/auth/cloud-platform"]

serviceAccount string

Outputs

state string

Possible Values

STATE_UNSPECIFIEDPENDINGRUNNINGCANCELLINGCANCELLEDSUCCEEDEDFAILEDUNRECOGNIZED

Core Plugins and tasks

Tasks that provide Kestra's built-in orchestration, I/O, and observability capabilities.

CoreStorageScript

PostgreSQL

Tasks that connect to PostgreSQL via JDBC to run queries and loads.

Database

SurrealDB

Tasks that run SurrealQL queries against SurrealDB.

Database

SparkSubmit

More Plugins in this Category

Core Plugins and tasks

PostgreSQL

SurrealDB

2.0.0