Run a Microsoft Fabric Data Pipeline

Trigger and monitor a Microsoft Fabric Data Pipeline directly from Kestra. This blueprint authenticates to Fabric with a service principal, submits a pipeline run with runtime parameters, and blocks until the job reaches a terminal state. It closes the gap between Fabric's own scheduling and the wider data platform: instead of running Fabric pipelines in isolation, you orchestrate them alongside ingestion, transformation, reporting, and alerting from one declarative workflow.

How it works

1. The single run_pipeline task uses io.kestra.plugin.microsoft.fabric.data.engineering.RunPipeline to call the Fabric REST API.
2. It authenticates with a service principal: tenantId, clientId, and clientSecret are read from Kestra secrets.
3. It targets a specific pipeline via workspaceId and pipelineId, both wired to the workspace_id and pipeline_id flow variables.
4. Runtime parameters pass environment and a templated date ({{ now() | dateFormat('yyyy-MM-dd') }}) into the pipeline run.
5. With wait: true, the task polls every pollFrequency (PT10S) until the run completes, fails, is cancelled, or is deduped, bounded by a timeout of PT2H.

What you get

• A native Fabric pipeline trigger with no glue scripts or custom REST code.
• Dynamic, per-run parameters injected from Kestra expressions.
• Synchronous execution: downstream tasks only run once the pipeline finishes.
• Final run status and identifiers available as task outputs for branching, reporting, or notifications.

Who it's for

• Data engineers consolidating Fabric pipelines into a central orchestrator.
• Analytics teams chaining Fabric runs with upstream prep and downstream BI refreshes.
• Platform teams standardizing on declarative, version-controlled pipelines.

Why orchestrate this with Kestra

Fabric's built-in scheduler runs pipelines on a clock, but it cannot react to external events, retry across tools, or stitch a Fabric run into an end-to-end lineage. With Kestra you add event and schedule triggers, automatic retries, full execution lineage across systems, and declarative YAML that lives in Git. You can gate the Fabric run on upstream data landing, then fan out to notifications or BI refreshes, all in one place.

Prerequisites

• A Microsoft Fabric workspace with a Data Pipeline.
• An Entra ID service principal granted access to the workspace.
• The workspace and pipeline GUIDs from the Fabric portal.

Secrets

• FABRIC_TENANT_ID: Entra ID tenant ID.
• FABRIC_CLIENT_ID: service principal client ID.
• FABRIC_CLIENT_SECRET: service principal client secret.

Quick start

1. Store FABRIC_TENANT_ID, FABRIC_CLIENT_ID, and FABRIC_CLIENT_SECRET as Kestra secrets.
2. Replace the workspace_id and pipeline_id variables with your actual Fabric GUIDs.
3. Adjust the parameters map to match the parameters your pipeline expects.
4. Run the flow and watch it poll until the pipeline reaches a terminal state.

How to extend

• Add a schedule or event trigger to launch the pipeline on data arrival instead of a fixed clock.
• Set wait: false for fire-and-forget runs when you do not need to block.
• Tune pollFrequency and timeout to match typical run duration.
• Chain downstream tasks that branch on the final status, send Slack or email alerts, or refresh a Power BI dataset.

Links

• plugin-microsoft documentation
• Microsoft Fabric orchestration with Kestra