Microsoft Fabric RunSparkJob

Microsoft Fabric RunSparkJob

Certified

Run a Microsoft Fabric Spark Job Definition

Submits a Fabric Spark Job Definition using the item job scheduler API and optionally waits for completion. Defaults: wait=true, pollFrequency=PT5S, timeout=PT1H.

yaml
type: io.kestra.plugin.microsoft.fabric.data.engineering.RunSparkJob
yaml
id: fabric_run_spark_job
namespace: company.team

tasks:
  - id: run_spark_job
    type: io.kestra.plugin.microsoft.fabric.data.engineering.RunSparkJob
    tenantId: "{{ secret('FABRIC_TENANT_ID') }}"
    clientId: "{{ secret('FABRIC_CLIENT_ID') }}"
    clientSecret: "{{ secret('FABRIC_CLIENT_SECRET') }}"
    workspaceId: "your-workspace-id"
    sparkJobDefinitionId: "your-spark-job-definition-id"
Properties

Spark Job Definition ID

Microsoft Fabric Spark Job Definition item GUID within the workspace

Workspace ID

Microsoft Fabric workspace GUID

Client ID

Service principal client (application) ID

Client Secret

Service principal client secret. When both clientId and clientSecret are provided, service principal authentication is used; otherwise DefaultAzureCredential is used.

Default{}

Execution parameters

Key/value parameters passed to the job execution

Reference (ref) of the pluginDefaults to apply to this task.

DefaultPT5S

Poll frequency

Interval between status checks when wait=true

Tenant ID

Azure Active Directory tenant ID for authentication

DefaultPT1H

Timeout

Maximum time to wait for completion; throws TimeoutException when exceeded

Defaulttrue

Wait for completion

Poll until the job reaches a terminal state; returns jobInstanceId immediately when false

Job instance ID

Unique ID of the Spark job instance

Final status

Terminal status of the Spark job

More Plugins in this Category

Clickhouse
Clickhouse

Tasks that connect to ClickHouse via JDBC to run queries and loads.

Data

5 tasks

2 blueprints

Weaviate
Weaviate

Tasks that manage and query vectors in Weaviate.

DataAI

4 tasks

2 blueprints