New to Kestra?
Use blueprints to kickstart your first workflows.
Download a Parquet file from Databricks DBFS and process it with Python
Pull a Parquet file from Databricks File System (DBFS) into Kestra internal storage and process it with a Python Pandas script running in Docker.
Categories
id: download-parquet-from-databricks
namespace: company.team
description: >
This flow will download a Parquet file from Databricks File System (DBFS) to
Kestra's internal storage.
tasks:
- id: download
type: io.kestra.plugin.databricks.dbfs.Download
authentication:
token: "{{ secret('DATABRICKS_TOKEN') }}"
host: "{{ secret('DATABRICKS_HOST') }}"
from: /Shared/myFile.parquet
- id: process_downloaded_file
type: io.kestra.plugin.scripts.python.Script
taskRunner:
type: io.kestra.plugin.scripts.runner.docker.Docker
dependencies:
- pandas
script: |
import pandas as pd
df = pd.read_parquet("{{ outputs.download.uri }}")
df.head()
Pull a Parquet file out of the Databricks File System (DBFS) and into Kestra so you can validate, profile, or transform it without staying locked inside the Databricks workspace. This blueprint downloads a Databricks-generated dataset into Kestra internal storage and immediately loads it into a Python Pandas script, giving you a portable, reusable pattern for moving Databricks data into the rest of your pipeline.
How it works
- The
downloadtask usesio.kestra.plugin.databricks.dbfs.Downloadto fetch a Parquet file from a DBFS path (from: /Shared/myFile.parquet), authenticating with a token and host. The downloaded file lands in Kestra internal storage and exposes its location throughoutputs.download.uri. - The
process_downloaded_filetask is anio.kestra.plugin.scripts.python.Scriptthat runs on theio.kestra.plugin.scripts.runner.docker.Dockertask runner withpandasdeclared as a dependency. It reads the file withpd.read_parquet("{{ outputs.download.uri }}")and callsdf.head()to inspect the dataset.
What you get
- A Databricks Parquet file copied into Kestra internal storage on demand.
- A reproducible Python and Pandas environment provisioned in Docker, no local setup required.
- A clean handoff via
outputs.download.urithat any downstream task can consume. - A starting point for data quality checks, exploratory analysis, or further ETL.
Who it's for
- Data engineers who need Databricks outputs available to non-Databricks tooling.
- Analytics engineers running validation or profiling on Lakehouse datasets.
- Platform teams centralizing data movement and orchestration in one place.
Why orchestrate this with Kestra
Databricks jobs schedule work inside the Databricks workspace, but they do not easily reach across your wider stack. With Kestra you express the whole flow as declarative YAML, add event or schedule triggers, attach retries to the download, and capture execution lineage and outputs so each run is auditable. The Parquet file becomes a first-class artifact in Kestra internal storage that downstream tasks, in any language or tool, can pick up, something the Databricks scheduler alone cannot coordinate.
Prerequisites
- A Databricks workspace with a Parquet file in DBFS.
- A Kestra instance with Docker available for the Python task runner.
Secrets
DATABRICKS_TOKEN: a Databricks personal access token used for authentication.DATABRICKS_HOST: the Databricks workspace host URL.
Quick start
- Add the
DATABRICKS_TOKENandDATABRICKS_HOSTsecrets to your Kestra instance. - Update the
frompath to point at your own DBFS Parquet file. - Execute the flow and confirm the file downloads to Kestra internal storage.
- Review the Python task logs to see the
df.head()preview.
How to extend
- Replace
df.head()with real transformations, aggregations, or quality assertions. - Push the processed data onward to a warehouse, object store, or database.
- Add a schedule or event trigger to run the flow automatically.
- Loop over multiple DBFS paths to download a batch of files.
Links
More Data Blueprints
