Azure SparkBatchJobCreate
CertifiedSubmit Spark batch to Synapse pool
Azure SparkBatchJobCreate
Submit Spark batch to Synapse pool
Creates and submits a Spark batch job to an Azure Synapse Analytics Spark pool using the workspace endpoint, pool name, and optional resource overrides. Requires valid Azure credentials with permission to submit Spark jobs.
type: io.kestra.plugin.azure.synapse.SparkBatchJobCreateExamples
id: azure_synapse_spark_batch
namespace: company.team
tasks:
- id: spark_job
type: io.kestra.plugin.azure.synapse.SparkBatchJobCreate
endpoint: "https://myworkspace.dev.azuresynapse.net"
sparkPoolName: "mysparkpool"
tenantId: "{{ secret('AZURE_TENANT_ID') }}"
clientId: "{{ secret('AZURE_CLIENT_ID') }}"
clientSecret: "{{ secret('AZURE_CLIENT_SECRET') }}"
name: "WordCount"
file: "abfss://data@mystorage.dfs.core.windows.net/jars/wordcount.jar"
className: "org.example.WordCount"
arguments:
- "abfss://data@mystorage.dfs.core.windows.net/input/shakespeare.txt"
- "abfss://data@mystorage.dfs.core.windows.net/output/"
driverMemory: "28g"
driverCores: 4
executorMemory: "28g"
executorCores: 4
executorCount: 2
Properties
endpoint *Requiredstring
Synapse workspace endpoint
Workspace URL in the form https://{workspace}.dev.azuresynapse.net.
file *Requiredstring
Main application file
Path to the main application file (JAR, Python, etc.) in ADLS Gen2 storage in the form abfss://{container}@{storage-account}.dfs.core.windows.net/path/to/file.
name *Requiredstring
Spark batch job name
sparkPoolName *Requiredstring
Spark pool name
Name of the Spark pool where the batch job will be submitted.
tenantId *Requiredstring
Azure AD tenant ID (GUID)
archives array
Archives
Archives to be used in the job.
arguments array
Command line arguments
Arguments passed to the main method.
className string
Main class name
Fully qualified class name for Java/Scala Spark jobs.
clientId string
Client ID of the Azure AD application
Application (client) ID used for service principal authentication.
clientSecret string
Client secret for the Azure AD application
Secret value associated with the client ID; store in a Kestra secret.
conf object
Spark configuration
Spark configuration properties for the job.
driverCores integerstring
Driver cores
Number of cores for the driver process.
driverMemory string
Driver memory
Memory for the driver process (e.g., 28g).
executorCores integerstring
Executor cores
Number of cores per executor.
executorCount integerstring
Executor count
Number of executor processes to launch.
executorMemory string
Executor memory
Memory per executor process (e.g., 28g).
files array
Additional files
Additional files to be used in the job.
jars array
Additional JAR files
Additional JARs to be used in the job.
pemCertificate string
PEM-encoded certificate content for client authentication
PEM text for certificate-based auth; alternative to clientSecret.
pluginDefaultsRef Non-dynamicstring
Reference (ref) of the pluginDefaults to apply to this task.
pyFiles array
Additional Python files
Additional Python files to be used in the job.
Outputs
appId string
Spark application ID
jobId integer
Spark batch job ID
state string
Spark batch job state
Possible states: not_started, starting, running, idle, busy, shutting_down, error, dead, killed, success.
