Azure SparkBatchJobCreate | Kestra

Azure SparkBatchJobCreate
Certified
Submit Spark batch to Synapse pool

Creates and submits a Spark batch job to an Azure Synapse Analytics Spark pool using the workspace endpoint, pool name, and optional resource overrides. Requires valid Azure credentials with permission to submit Spark jobs.

yaml
type: io.kestra.plugin.azure.synapse.SparkBatchJobCreate

Examples

yaml
id: azure_synapse_spark_batch
namespace: company.team

tasks:
  - id: spark_job
    type: io.kestra.plugin.azure.synapse.SparkBatchJobCreate
    endpoint: "https://myworkspace.dev.azuresynapse.net"
    sparkPoolName: "mysparkpool"
    tenantId: "{{ secret('AZURE_TENANT_ID') }}"
    clientId: "{{ secret('AZURE_CLIENT_ID') }}"
    clientSecret: "{{ secret('AZURE_CLIENT_SECRET') }}"
    name: "WordCount"
    file: "abfss://data@mystorage.dfs.core.windows.net/jars/wordcount.jar"
    className: "org.example.WordCount"
    arguments:
      - "abfss://data@mystorage.dfs.core.windows.net/input/shakespeare.txt"
      - "abfss://data@mystorage.dfs.core.windows.net/output/"
    driverMemory: "28g"
    driverCores: 4
    executorMemory: "28g"
    executorCores: 4
    executorCount: 2

Properties

endpoint *string

Synapse workspace endpoint

Workspace URL in the form https://{workspace}.dev.azuresynapse.net.

file *string

Main application file

Path to the main application file (JAR, Python, etc.) in ADLS Gen2 storage in the form abfss://{container}@{storage-account}.dfs.core.windows.net/path/to/file.

name *string

Spark batch job name

sparkPoolName *string

Spark pool name

Name of the Spark pool where the batch job will be submitted.

tenantId *string

Azure AD tenant ID (GUID)

archives array

SubTypestring

Archives

Archives to be used in the job.

arguments array

SubTypestring

Command line arguments

Arguments passed to the main method.

className string

Main class name

Fully qualified class name for Java/Scala Spark jobs.

clientId string

Client ID of the Azure AD application

Application (client) ID used for service principal authentication.

clientSecret string

Client secret for the Azure AD application

Secret value associated with the client ID; store in a Kestra secret.

conf object

Spark configuration

Spark configuration properties for the job.

driverCores integerstring

Driver cores

Number of cores for the driver process.

driverMemory string

Driver memory

Memory for the driver process (e.g., 28g).

executorCores integerstring

Executor cores

Number of cores per executor.

executorCount integerstring

Executor count

Number of executor processes to launch.

executorMemory string

Executor memory

Memory per executor process (e.g., 28g).

files array

SubTypestring

Additional files

Additional files to be used in the job.

jars array

SubTypestring

Additional JAR files

Additional JARs to be used in the job.

pemCertificate string

PEM-encoded certificate content for client authentication

PEM text for certificate-based auth; alternative to clientSecret.

pluginDefaultsRef string

Reference (ref) of the pluginDefaults to apply to this task.

pyFiles array

SubTypestring

Additional Python files

Additional Python files to be used in the job.

tags object

Tags

Tags to associate with the job.

Outputs

appId string

Spark application ID

jobId integer

Spark batch job ID

state string

Spark batch job state

Possible states: not_started, starting, running, idle, busy, shutting_down, error, dead, killed, success.

Liquibase

Tasks that run Liquibase change management commands.

OPA (Open Policy Agent)

Tasks to manage Open Policy Agent policies and decisions, including upload, evaluation, and partial evaluation.