Fabric runs the notebooks, pipelines, and lakehouse. Kestra decides when each job fires, what runs next, and what happens beyond Fabric. Trigger on any event, wait for completion, layer OneLake bronze to gold, and chain with dbt, on-prem databases, and the rest of your stack in one execution history.
Fabric's built-in scheduler fires notebooks and pipelines on a cron inside the Fabric workspace. Kestra runs above it: it triggers RunPipeline, RunNotebook, and RunSparkJob the moment an upstream event lands, waits for terminal status, layers OneLake bronze to gold, and chains Fabric with dbt, Kafka, and on-prem systems that the Fabric scheduler cannot reach. One execution history spans the whole pipeline, inside Fabric and beyond it.
Fabric runs the notebooks, pipelines, Spark, and OneLake. Kestra runs the steps around them: what triggers each job, what runs next, who is alerted, and where the cross-tool audit trail lives.
Fabric's scheduler fires items on a fixed cadence inside the workspace. Kestra triggers io.kestra.plugin.microsoft.fabric.data.engineering.RunPipeline the moment an upstream event lands: a webhook, an Azure Blob arrival, a Kafka message, or a schedule. The Fabric job runs on the event, not on the next workspace tick.
RunNotebook, RunPipeline, and RunSparkJob each submit a Fabric item job and wait for completion by default (polling every 5s). Each becomes one step in a larger Kestra flow, so a notebook can be bracketed by the load before it and the activation after it, with per-step retries.
The OneLake tasks (onelake.Upload, Download, Delete) move files in and out of a Lakehouse over the ADLS Gen2 API. Kestra layers raw into bronze, conforms to silver, and aggregates into gold as ordered steps, each retrying independently, with the whole refresh under one execution ID.
warehouse.Query runs SQL against a Fabric Warehouse over JDBC with service-principal auth, returning rows or writing results to internal storage. A query result can gate the next step, feed a notebook, or branch the flow, so the warehouse is part of the pipeline, not a dead end.
Fabric orchestrates within the Microsoft estate. Kestra chains a Fabric pipeline with a dbt build, an on-prem database extract, a warehouse load, or a Kafka event in one flow, so Fabric is one step in a pipeline that spans the whole stack, not an island.
Kestra runs self-hosted on Docker or Kubernetes, or air-gapped, and is not bound to the Microsoft cloud. It orchestrates Fabric alongside other clouds and on-prem from one control plane, which matters for regulated workloads that cannot be driven from a single vendor's scheduler.
Patterns cloud data teams run in production today. Each one shows the flow end to end, with the real plugin classes in play.
A webhook (or Azure Blob arrival, or Kafka message) fires the flow, Kestra runs RunPipeline and waits for terminal status, then branches on the result. The Fabric pipeline runs when the source is confirmed ready, not on a workspace schedule.
RunPipeline polls the job instance until terminal, failing on Failed or Cancelled.
The pipeline runs on a webhook, file, or message, not the next cron tick.
An If task routes success and failure to different paths.
Kestra runs a Fabric RunNotebook job, waits for it, then triggers a dbt build on the output and confirms on Slack. The notebook is one step in a chain it could not express inside the workspace alone.
RunNotebook submits the item job and waits for completion.
The notebook output feeds dbt, a warehouse load, or an on-prem system.
A dbt retry never re-runs the notebook.
The medallion pattern on OneLake. Kestra downloads raw files from bronze with onelake.Download, runs a Spark job to conform to silver, then uploads aggregated gold with onelake.Upload. Each hop is a step with its own retry.
Bronze, silver, and gold hops each retry independently.
RunSparkJob handles the conform-and-aggregate work.
The whole medallion run shares one execution ID.
Kestra runs a warehouse.Query against the Fabric Warehouse, branches on the result with an If task, and only proceeds when a freshness or row-count check passes. A failed check halts the flow and pages PagerDuty before downstream runs.
A warehouse.Query result decides whether downstream runs.
JDBC with Active Directory service principal, no interactive login.
A failed check pages the on-call before anything downstream runs.
Kestra extracts from an on-prem database, uploads the result to OneLake with onelake.Upload, then runs a Fabric pipeline on it, all in one flow with one execution ID. The pattern brings data Fabric cannot reach natively into the Lakehouse.
On-prem databases and file shares land in OneLake through one flow.
The extract, the OneLake upload, and the Fabric run share one execution ID.
A Fabric retry never re-runs the on-prem extract.
| Capability | | Fabric Scheduler | | |
|---|---|---|---|---|
| Trigger on webhook, Blob, Kafka, schedule | Native event triggers | Workspace cron only | Event Grid wiring required | Job schedules, file triggers |
| Chain Fabric with dbt, on-prem, the warehouse | Native plugins, any target | Fabric-centric | Azure-centric, limited | Databricks-centric |
| OneLake medallion as ordered steps | Download + Spark + Upload | Native, Fabric-only | Dataflows, Azure-bound | Native, Databricks-only |
| Run fully self-hosted, off the cloud | Docker, Kubernetes, air-gapped | SaaS only | Azure-managed only | Managed only |
| Reach beyond the Microsoft estate | Fabric + on-prem + any tool in one flow | Microsoft only | Azure only | Limited |
| Self-service runs for non-engineers | Kestra Apps | Workspace access required | No native form layer | No native form layer |
| Declarative YAML, IaC-friendly | YAML + Terraform provider | GUI-first | JSON pipelines / portal | Notebooks + UI |
Find answers to your questions right here, and don't hesitate to Contact Us if you couldn't find what you're looking for.
Trigger notebooks and pipelines on any event, layer OneLake bronze to gold, query the warehouse, and chain Fabric with the rest of your stack. Open source, self-hosted, event-driven.
