ESC
Ask Kestra AI
ESC
Kestra AI

Hi! I'm your Kestra AI assistant.
Ask me anything about workflows.

EXAMPLE QUESTIONS

How to sync my flows with Git?
How to write expression for previous tasks outputs?
How to run Python script?

Azure SparkBatchJobCreate

Azure SparkBatchJobCreate

Certified

Submit Spark batch to Synapse pool

Creates and submits a Spark batch job to an Azure Synapse Analytics Spark pool using the workspace endpoint, pool name, and optional resource overrides. Requires valid Azure credentials with permission to submit Spark jobs.

yaml
type: io.kestra.plugin.azure.synapse.SparkBatchJobCreate
yaml
id: azure_synapse_spark_batch
namespace: company.team

tasks:
  - id: spark_job
    type: io.kestra.plugin.azure.synapse.SparkBatchJobCreate
    endpoint: "https://myworkspace.dev.azuresynapse.net"
    sparkPoolName: "mysparkpool"
    tenantId: "{{ secret('AZURE_TENANT_ID') }}"
    clientId: "{{ secret('AZURE_CLIENT_ID') }}"
    clientSecret: "{{ secret('AZURE_CLIENT_SECRET') }}"
    name: "WordCount"
    file: "abfss://data@mystorage.dfs.core.windows.net/jars/wordcount.jar"
    className: "org.example.WordCount"
    arguments:
      - "abfss://data@mystorage.dfs.core.windows.net/input/shakespeare.txt"
      - "abfss://data@mystorage.dfs.core.windows.net/output/"
    driverMemory: "28g"
    driverCores: 4
    executorMemory: "28g"
    executorCores: 4
    executorCount: 2
Properties

Synapse workspace endpoint

Workspace URL in the form https://{workspace}.dev.azuresynapse.net.

Main application file

Path to the main application file (JAR, Python, etc.) in ADLS Gen2 storage in the form abfss://{container}@{storage-account}.dfs.core.windows.net/path/to/file.

Spark batch job name

Spark pool name

Name of the Spark pool where the batch job will be submitted.

Azure AD tenant ID (GUID)

SubTypestring

Archives

Archives to be used in the job.

SubTypestring

Command line arguments

Arguments passed to the main method.

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions
assetFailureBehaviorstring
Possible Values
IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

id*string
Min length1
typestring
outputs

The assets produced as outputs.

id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*string
Min length1

Custom asset type

descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150

Main class name

Fully qualified class name for Java/Scala Spark jobs.

Client ID of the Azure AD application

Application (client) ID used for service principal authentication.

Client secret for the Azure AD application

Secret value associated with the client ID; store in a Kestra secret.

Spark configuration

Spark configuration properties for the job.

Driver cores

Number of cores for the driver process.

Driver memory

Memory for the driver process (e.g., 28g).

Executor cores

Number of cores per executor.

Executor count

Number of executor processes to launch.

Executor memory

Memory per executor process (e.g., 28g).

SubTypestring

Additional files

Additional files to be used in the job.

SubTypestring

Additional JAR files

Additional JARs to be used in the job.

PEM-encoded certificate content for client authentication

PEM text for certificate-based auth; alternative to clientSecret.

SubTypestring

Additional Python files

Additional Python files to be used in the job.

Tags

Tags to associate with the job.

Spark application ID

Spark batch job ID

Spark batch job state

Possible states: not_started, starting, running, idle, busy, shutting_down, error, dead, killed, success.

More Plugins in this Category

Ashby
Ashby

Ashby plugin for Kestra

Data

1 tasks

1 blueprints

SAP HANA
SAP HANA

Tasks that query SAP HANA using the official SAP HANA JDBC driver (ngdbc.jar).

Data

3 tasks

3 blueprints