SparkBatchJobCreate

yaml
type: "io.kestra.plugin.azure.synapse.SparkBatchJobCreate"

Examples

yaml
id: azure_synapse_spark_batch
namespace: company.team

tasks:
  - id: spark_job
    type: io.kestra.plugin.azure.synapse.SparkBatchJobCreate
    endpoint: "https://myworkspace.dev.azuresynapse.net"
    sparkPoolName: "mysparkpool"
    tenantId: "{{ secret('AZURE_TENANT_ID') }}"
    clientId: "{{ secret('AZURE_CLIENT_ID') }}"
    clientSecret: "{{ secret('AZURE_CLIENT_SECRET') }}"
    name: "WordCount"
    file: "abfss://data@mystorage.dfs.core.windows.net/jars/wordcount.jar"
    className: "org.example.WordCount"
    arguments:
      - "abfss://data@mystorage.dfs.core.windows.net/input/shakespeare.txt"
      - "abfss://data@mystorage.dfs.core.windows.net/output/"
    driverMemory: "28g"
    driverCores: 4
    executorMemory: "28g"
    executorCores: 4
    executorCount: 2

Properties

endpoint *string

file *string

name *string

sparkPoolName *string

tenantId *string

archives array

SubTypestring

arguments array

SubTypestring

className string

clientId string

clientSecret string

conf object

SubTypestring

driverCores integerstring

driverMemory string

executorCores integerstring

executorCount integerstring

executorMemory string

files array

SubTypestring

jars array

SubTypestring

pemCertificate string

pyFiles array

SubTypestring

tags object

SubTypestring

Outputs

appId string

jobId integer

state string

Core Plugins and tasks

Tasks that provide Kestra's built-in orchestration, I/O, and observability capabilities.

CoreStorageScript

PostgreSQL

Tasks that connect to PostgreSQL via JDBC to run queries and loads.

Database

SurrealDB

Tasks that run SurrealQL queries against SurrealDB.

Database

SparkBatchJobCreate

More Plugins in this Category

Core Plugins and tasks

PostgreSQL

SurrealDB

1.5.0