Run a Microsoft Fabric Spark Job

Run a Microsoft Fabric Spark Job Definition directly from a Kestra flow and block until it reaches a terminal state. This blueprint solves a common gap in the Fabric platform: triggering Spark batch workloads as a managed, observable step inside a larger data pipeline instead of launching them by hand in the Fabric portal or stitching them together with custom scripts. It authenticates with a service principal, submits the job, polls for status, and surfaces the outcome so the rest of your orchestration can react to it.

How it works

The flow runs a single run_spark_job task of type io.kestra.plugin.microsoft.fabric.data.engineering.RunSparkJob:

1. Authenticates to Microsoft Fabric with a service principal using tenantId, clientId, and clientSecret, all read from Kestra secrets.
2. Submits the Spark Job Definition identified by workspaceId and sparkJobDefinitionId, both pulled from flow variables.
3. Waits for the job because wait: true, polling status every PT30S (pollFrequency) until the job reaches a terminal state.
4. Enforces a timeout of PT4H so a stuck or runaway job cannot block the flow indefinitely.

What you get

• A managed Fabric Spark job submission that runs as a first-class pipeline step.
• Synchronous execution: the flow waits for success or failure rather than firing and forgetting.
• Job status and instance outputs you can branch on or pass to downstream tasks.
• A hard timeout boundary that protects the rest of the pipeline.

Who it's for

• Data engineers running Spark batch processing on Microsoft Fabric.
• Platform teams that need Fabric jobs embedded in end-to-end pipelines.
• Analytics teams chaining Spark transformations with ingestion and reporting.

Why orchestrate this with Kestra

Fabric's built-in scheduler can run a Spark job on a clock, but it cannot easily make that job one dependent step in a cross-system pipeline. With Kestra you trigger the job on real events (a file landing, an upstream load finishing, an API webhook), add automatic retries and a timeout, capture lineage and full execution logs, and express the whole pipeline as declarative, version-controlled YAML. The Spark job becomes a coordinated link in a larger flow rather than an isolated scheduled task.

Prerequisites

• A Microsoft Fabric workspace with a Spark Job Definition.
• An Entra ID service principal with permission to run Fabric items.
• The workspace_id and spark_job_definition_id GUIDs from the Fabric portal.

Secrets

• FABRIC_TENANT_ID: Entra ID tenant GUID for the service principal.
• FABRIC_CLIENT_ID: service principal (application) client ID.
• FABRIC_CLIENT_SECRET: service principal client secret.

Quick start

1. Store FABRIC_TENANT_ID, FABRIC_CLIENT_ID, and FABRIC_CLIENT_SECRET as Kestra secrets.
2. Set the workspace_id and spark_job_definition_id variables to your Fabric GUIDs.
3. Adjust pollFrequency and timeout to match your workload's expected runtime.
4. Run the flow and watch the Spark job execute and report status in the Kestra UI.

How to extend

• Add a trigger so the job fires on a schedule or in response to an upstream event.
• Chain downstream tasks (notifications, reporting, data quality checks) off the job status.
• Add tasks under errors: to alert on failure via Slack or email.
• Add a retry policy to run_spark_job for transient Fabric or network issues.

Links

• Plugin docs: https://kestra.io/plugins/plugin-microsoft-fabric
• Orchestration: https://kestra.io/orchestration/microsoft-fabric