Google Cloud SparkSqlSubmit

Google Cloud SparkSqlSubmit

Certified

Submit Spark SQL batch to Dataproc

Executes Spark SQL from a script file; supports extra JARs for UDFs.

yaml
type: io.kestra.plugin.gcp.dataproc.batches.SparkSqlSubmit
yaml
id: gcp_dataproc_spark_sql_submit
namespace: company.team
tasks:
  - id: spark_sql_submit
    type: io.kestra.plugin.gcp.dataproc.batches.SparkSqlSubmit
    queryFileUri: 'gs://spark-jobs-kestra/foobar.sql'
    name: test-sparksql
    region: europe-west3
Properties

Batch name

Base name; a slugified suffix with execution id is appended and truncated to 63 chars

Query file URI

HCFS URI to the Spark SQL script (gs://, hdfs://, or file://)

Dataproc region

Region endpoint used for the batch controller

SubTypestring

Archive URIs

HCFS URIs of archives extracted into each executor dir (.jar, .tar, .tar.gz, .tgz, .zip)

SubTypestring

Driver arguments

Arguments passed to the driver; avoid options that belong in batch properties (e.g., --conf)

Execution configuration

Network, service account, and KMS settings for the workload

Definitions
kmsKeystring

KMS key

Cloud KMS key for encryption

networkTagsarray
SubTypestring

Network tags

Applied to workload for network control

networkUristring

Network URI

VPC network URI for the workload

serviceAccountEmailstring

Service account

Email of the service account used to run the batch

subnetworkUristring

Subnetwork URI

Optional subnetwork URI for the workload

SubTypestring

File URIs

HCFS URIs copied into each executor working dir (gs://, hdfs://, or file://)

The GCP service account to impersonate

SubTypestring

Jar URIs

HCFS URIs of JARs added to driver and executor classpaths

Peripherals configuration

Optional Metastore and Spark History Server settings

Definitions
metastoreServicestring

Metastore service

Resource name of an existing Dataproc Metastore service (projects/{project}/locations/{region}/services/{id})

sparkHistoryServer

Spark History Server

Optional Dataproc cluster to serve Spark History for the workload

dataprocClusterstring

Spark History Server cluster

Dataproc cluster resource name (projects/{project}/regions/{region}/clusters/{name})

Reference (ref) of the pluginDefaults to apply to this task.

The GCP project ID

Runtime configuration

Container image, runtime version, and properties

Definitions
containerImagestring

Container image

Optional custom runtime image; defaults to Dataproc image when unset

propertiesobject

Runtime properties

Key/value pairs passed to the runtime

versionstring

Runtime version

Optional Dataproc runtime version string

SubTypestring
Default["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used

The GCP service account

Possible Values
STATE_UNSPECIFIEDPENDINGRUNNINGCANCELLINGCANCELLEDSUCCEEDEDFAILEDUNRECOGNIZED

Batch state

More Plugins in this Category

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks

PostgreSQL
PostgreSQL

Tasks that connect to PostgreSQL via JDBC to run queries and loads.

Data

6 tasks

25 blueprints