Download from OneLake, Process with Python, and Upload Back

Read a file from a Microsoft Fabric Lakehouse via OneLake, transform it with a Python script, and write the result back to the same Lakehouse, all as one orchestrated pipeline. This blueprint solves a common Fabric gap: you have data sitting in OneLake but need custom Python logic (Pandas, PyArrow, scikit-learn, anything pip-installable) that does not fit cleanly into Spark notebooks or pipelines, and you want it scheduled, retried, and observable alongside the rest of your data stack.

How it works

1. download_from_onelake uses io.kestra.plugin.microsoft.fabric.onelake.Download to pull a Parquet file from the Lakehouse Files/ directory into Kestra internal storage. It authenticates with a service principal (tenantId, clientId, clientSecret) and targets the data with workspaceId, itemId, and filePath.
2. process runs io.kestra.plugin.scripts.python.Script inside a Docker taskRunner, pulling in pandas and pyarrow via dependencies. It reads the downloaded Parquet via inputFiles, builds a summary with df.describe(), and emits report.csv through outputFiles.
3. upload_report uses io.kestra.plugin.microsoft.fabric.onelake.Upload to write the CSV back to OneLake under a date-partitioned filePath (reports/{{ now() | dateFormat('yyyy-MM-dd') }}/report.csv).

What you get

• A round-trip OneLake read, transform, write pipeline with no glue code.
• Isolated, reproducible Python execution in Docker with pinned dependencies.
• Date-partitioned output written straight back into the Lakehouse.
• Full run history, logs, and outputs for every step in the Kestra UI.

Who it's for

• Data engineers running lightweight ETL on Fabric-managed datasets.
• Analytics teams generating recurring summary or quality reports from OneLake.
• ML practitioners who need custom Python on Lakehouse data outside Spark.

Why orchestrate this with Kestra

Fabric pipelines and notebooks schedule work inside Fabric, but they cannot easily fan out to arbitrary Python, react to upstream events from other systems, or sit in a wider cross-tool DAG. Kestra gives you declarative YAML, event triggers (file arrival, webhooks, flow dependencies), automatic retries on transient OneLake or auth failures, and end-to-end lineage across Fabric and everything else in your stack, with the transform running in a clean, version-pinned container.

Prerequisites

• A Microsoft Fabric workspace and Lakehouse with a service principal granted access.
• The workspace and Lakehouse GUIDs.
• Docker available to the Kestra worker for the Python task runner.

Secrets

• FABRIC_TENANT_ID: Entra tenant ID for the service principal.
• FABRIC_CLIENT_ID: service principal application (client) ID.
• FABRIC_CLIENT_SECRET: service principal client secret.

Quick start

1. Store FABRIC_TENANT_ID, FABRIC_CLIENT_ID, and FABRIC_CLIENT_SECRET as Kestra secrets.
2. Set the workspace_id and lakehouse_id variables to your Fabric GUIDs.
3. Point input_path at the file inside the Lakehouse Files/ directory.
4. Execute the flow and check report.csv under the dated reports/ path in OneLake.

How to extend

• Swap the describe() summary for real feature engineering, validation, or model scoring.
• Add more dependencies (scikit-learn, duckdb, great-expectations) for richer logic.
• Add a Schedule or file-arrival trigger to run the pipeline automatically.
• Chain a notification task (Slack, email) to report on the uploaded result.

Links

• plugin-microsoft documentation
• Microsoft Fabric orchestration with Kestra