End-to-End Fabric Orchestration: Ingest, Transform, and Validate

This blueprint orchestrates a complete Microsoft Fabric data pipeline (raw ingestion, Notebook transformation, and Warehouse validation) from one declarative Kestra flow on a nightly schedule. It solves the common Fabric problem of stitching separate Data Pipelines, Notebooks, and Warehouse queries into a single dependency-aware workflow with retries, timeouts, and end-to-end observability, instead of chaining them with ad hoc scripts or manual reruns.

How it works

1. A io.kestra.plugin.core.trigger.Schedule trigger named nightly fires every day at 02:00 (cron: 0 2 * * *).
2. run_ingestion_pipeline (io.kestra.plugin.microsoft.fabric.data.engineering.RunPipeline) starts a Fabric Data Pipeline to load raw data into the Bronze Lakehouse layer. It passes load_date as a parameter, sets wait: true, and polls every PT15S with a PT1H timeout.
3. run_transform_notebook (io.kestra.plugin.microsoft.fabric.data.engineering.RunNotebook) runs a Fabric Notebook to clean and conform Bronze into the Silver layer, passing source_layer: bronze and target_layer: silver. It waits with a PT20S poll frequency and a PT2H timeout.
4. validate_results (io.kestra.plugin.microsoft.fabric.warehouse.Query) runs a FETCH_ONE SQL query against the Fabric Warehouse to count rows written to silver.fact_orders for the current load_date.
5. log_validation (io.kestra.plugin.core.log.Log) logs the row count returned by the validation query.
6. The errors block runs on_failure, an ERROR-level Log task that reports which task failed.

What you get

• A single nightly flow spanning three Fabric workloads (Pipeline, Notebook, Warehouse).
• A medallion-style Bronze to Silver flow with a built-in row-count data quality check.
• Per-task wait, pollFrequency, and timeout control so long-running Fabric jobs are tracked to completion.
• A structured failure path ready to swap for Slack or email alerting.

Who it's for

• Data engineers running batch pipelines across multiple Fabric workloads.
• Analytics teams implementing medallion (Bronze, Silver, Gold) architectures.
• Platform teams who want retries, timeouts, and observability for Fabric jobs without custom orchestration code.

Why orchestrate this with Kestra

Fabric's native scheduling triggers each item in isolation and cannot express cross-item dependencies, conditional validation, or a unified failure path across a Pipeline, a Notebook, and a Warehouse query. Kestra adds event and schedule triggers, automatic retries, per-task timeouts, full execution lineage across all three Fabric workloads, and a single declarative YAML definition you can version-control. The Warehouse validation step and the errors handler give you a data quality gate and alerting that Fabric's per-item scheduler does not provide.

Prerequisites

• A Microsoft Fabric workspace with a Data Pipeline, a Notebook, a Lakehouse, and a Warehouse.
• A Microsoft Entra service principal with access to those Fabric items.

Secrets

• FABRIC_TENANT_ID: Entra tenant ID for the service principal.
• FABRIC_CLIENT_ID: service principal application (client) ID.
• FABRIC_CLIENT_SECRET: service principal client secret.

Quick start

1. Store FABRIC_TENANT_ID, FABRIC_CLIENT_ID, and FABRIC_CLIENT_SECRET as Kestra secrets.
2. Replace the vars GUIDs (workspace_id, lakehouse_id, pipeline_id, notebook_id, sql_endpoint_id, warehouse_id) with your Fabric identifiers.
3. Adjust the validation sql to match your Silver table and schema.
4. Add the flow and run it manually, then let the nightly schedule take over.

How to extend

• Add a Gold-layer RunNotebook step after validation for aggregated marts.
• Replace the on_failure Log task with a Slack or email notification task.
• Branch on the validate_results row count to fail the run when zero rows load.
• Parameterize load_date from a flow input to support backfills.

Links

• plugin-microsoft docs
• Microsoft Fabric orchestration