Google Cloud SparkSqlSubmit
CertifiedSubmit Spark SQL batch to Dataproc
Google Cloud SparkSqlSubmit
Submit Spark SQL batch to Dataproc
Executes Spark SQL from a script file; supports extra JARs for UDFs.
type: io.kestra.plugin.gcp.dataproc.batches.SparkSqlSubmitExamples
id: gcp_dataproc_spark_sql_submit
namespace: company.team
tasks:
- id: spark_sql_submit
type: io.kestra.plugin.gcp.dataproc.batches.SparkSqlSubmit
queryFileUri: 'gs://spark-jobs-kestra/foobar.sql'
name: test-sparksql
region: europe-west3
Properties
name *string
Batch name
Base name; a slugified suffix with execution id is appended and truncated to 63 chars
queryFileUri *string
Query file URI
HCFS URI to the Spark SQL script (gs://, hdfs://, or file://)
region *string
Dataproc region
Region endpoint used for the batch controller
archiveUris array
Archive URIs
HCFS URIs of archives extracted into each executor dir (.jar, .tar, .tar.gz, .tgz, .zip)
args array
Driver arguments
Arguments passed to the driver; avoid options that belong in batch properties (e.g., --conf)
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150execution
Execution configuration
Network, service account, and KMS settings for the workload
io.kestra.plugin.gcp.dataproc.batches.AbstractBatch-ExecutionConfiguration
KMS key
Cloud KMS key for encryption
Network tags
Applied to workload for network control
Network URI
VPC network URI for the workload
Service account
Email of the service account used to run the batch
Subnetwork URI
Optional subnetwork URI for the workload
fileUris array
File URIs
HCFS URIs copied into each executor working dir (gs://, hdfs://, or file://)
impersonatedServiceAccount string
The GCP service account to impersonate
jarFileUris array
Jar URIs
HCFS URIs of JARs added to driver and executor classpaths
peripherals
Peripherals configuration
Optional Metastore and Spark History Server settings
io.kestra.plugin.gcp.dataproc.batches.AbstractBatch-PeripheralsConfiguration
Metastore service
Resource name of an existing Dataproc Metastore service (projects/{project}/locations/{region}/services/{id})
Spark History Server
Optional Dataproc cluster to serve Spark History for the workload
io.kestra.plugin.gcp.dataproc.batches.AbstractBatch-SparkHistoryServerConfiguration
Spark History Server cluster
Dataproc cluster resource name (projects/{project}/regions/{region}/clusters/{name})
projectId string
The GCP project ID
runtime
Runtime configuration
Container image, runtime version, and properties
io.kestra.plugin.gcp.dataproc.batches.AbstractBatch-RuntimeConfiguration
Container image
Optional custom runtime image; defaults to Dataproc image when unset
Runtime properties
Key/value pairs passed to the runtime
Runtime version
Optional Dataproc runtime version string
scopes array
["https://www.googleapis.com/auth/cloud-platform"]The GCP scopes to be used
serviceAccount string
The GCP service account
Outputs
state string
STATE_UNSPECIFIEDPENDINGRUNNINGCANCELLINGCANCELLEDSUCCEEDEDFAILEDUNRECOGNIZEDBatch state
