Run a Microsoft Fabric Notebook

Run a Microsoft Fabric notebook as an orchestrated step in your data pipeline. This blueprint authenticates to Fabric with an Entra ID service principal, submits a notebook job with runtime parameters, and waits for the run to reach a terminal state before the flow continues. It solves the common gap where a Fabric notebook needs to run with dynamic inputs (a processing date, a target table, model settings) and feed downstream tasks, instead of running in isolation on its own schedule.

How it works

• The run_notebook task (io.kestra.plugin.microsoft.fabric.data.engineering.RunNotebook) authenticates using tenantId, clientId, and clientSecret pulled from Kestra secrets.
• It targets a specific notebook via workspaceId and notebookId, both wired to the workspace_id and notebook_id flow variables so you can swap GUIDs without touching the task.
• Runtime values are passed under parameters: here input_date is rendered with {{ now() | dateFormat('yyyy-MM-dd') }} and output_table is set to silver.cleaned_events.
• With wait: true, the task polls the Fabric job at the pollFrequency interval (PT15S) until the notebook completes, fails, is cancelled, or is deduped, and gives up after the timeout (PT1H).

What you get

• A parameterized, repeatable notebook run instead of manual portal clicks.
• Synchronous completion: the flow only proceeds once the notebook actually finishes.
• Job outputs (job instance id and status) available to downstream tasks.
• Secret-based authentication with no credentials in the flow definition.

Who it's for

• Data engineers running Fabric notebooks for transformation, cleansing, or feature engineering.
• Analytics teams promoting data through bronze, silver, and gold layers in OneLake.
• Platform teams that need Fabric workloads embedded in larger, cross-tool pipelines.

Why orchestrate this with Kestra

Fabric's built-in scheduler can run a notebook on a clock, but it cannot easily start a notebook in response to an external event, pass values computed earlier in a pipeline, or coordinate the notebook with non-Fabric systems. With Kestra you get event and schedule triggers, automatic retries, end-to-end lineage across tasks, and a declarative YAML definition kept in version control. The notebook becomes one composable step alongside ingestion, dbt, alerting, and reverse ETL, all governed from a single place.

Prerequisites

• A Microsoft Fabric workspace and a notebook you can run.
• An Entra ID service principal with permission to run items in that workspace.
• The workspace and notebook GUIDs.

Secrets

• FABRIC_TENANT_ID: your Entra ID tenant id.
• FABRIC_CLIENT_ID: the service principal application (client) id.
• FABRIC_CLIENT_SECRET: the service principal client secret.

Quick start

1. Add FABRIC_TENANT_ID, FABRIC_CLIENT_ID, and FABRIC_CLIENT_SECRET as Kestra secrets.
2. Set the workspace_id and notebook_id variables to your Fabric GUIDs.
3. Adjust the parameters map to the inputs your notebook expects.
4. Tune pollFrequency and timeout to your notebook's typical runtime.
5. Execute the flow and watch the run complete in the Kestra UI.

How to extend

• Add a schedule or webhook trigger to run the notebook automatically.
• Chain multiple RunNotebook tasks to orchestrate a medallion (bronze, silver, gold) pipeline.
• Feed parameters from upstream tasks or flow inputs instead of static values.
• Add error tasks under errors: to post a Slack or email alert when a run fails.
• Retry transient failures by adding a retry policy to the task.

Links

• Plugin docs: https://kestra.io/plugins/plugin-microsoft
• Orchestration: https://kestra.io/orchestration/microsoft-fabric