Query Databricks SQL and process lakehouse data with Python and Pandas

Pull data out of your Databricks lakehouse with a SQL query, hand the results to Python, and analyze them with Pandas, all in one declarative pipeline. This blueprint connects Databricks SQL warehouses to your broader data stack so analytics engineers and data scientists can collaborate on the same automated, reproducible workflow instead of trading manual CSV exports.

How it works

1. The sql_query task (io.kestra.plugin.databricks.sql.Query) connects to a Databricks SQL warehouse using accessToken, host, and an httpPath, then runs the SQL statement (SELECT * FROM samples.nyctaxi.trips LIMIT 100) and stores the result set as an internal output.
2. The csv task (io.kestra.plugin.serdes.csv.IonToCsv) reads {{ outputs.sql_query.uri }} and converts Kestra's internal Ion format into a portable CSV file.
3. The pandas task (io.kestra.plugin.scripts.python.Script) installs the pandas dependency, loads the CSV with pd.read_csv, and inspects it with df.head(), ready for any transformation or analysis you add.

What you get

• A repeatable extract path from a Databricks lakehouse table to CSV.
• Clean separation between SQL execution and Python processing.
• Query results materialized as Kestra outputs you can pass downstream.
• A working Pandas starting point for analysis, validation, or feature prep.

Who it's for

• Analytics engineers running scheduled extracts from Databricks.
• Data scientists who want lakehouse data in a Pandas DataFrame.
• Data teams bridging SQL analytics with Python workflows.

Why orchestrate this with Kestra

Databricks Workflows can schedule notebooks and SQL inside Databricks, but Kestra orchestrates the steps that live outside it: format conversion, arbitrary Python with its own dependencies, and downstream handoffs. With Kestra you get event and schedule triggers, automatic retries on transient failures, full execution lineage across the SQL and Python steps, and a declarative YAML definition you can version control, all without coupling your orchestration to a single vendor's scheduler.

Prerequisites

• A Databricks workspace with a running SQL warehouse.
• The httpPath of that warehouse (Connection details tab).
• Python available on the worker for the script task.

Secrets

• DATABRICKS_TOKEN: a Databricks personal access token used as accessToken.
• DATABRICKS_HOST: your workspace host (for example, dbc-xxxx.cloud.databricks.com).

Quick start

1. Add the DATABRICKS_TOKEN and DATABRICKS_HOST secrets to your Kestra instance.
2. Replace the httpPath value with your own SQL warehouse path.
3. Swap the sample SELECT for a query against your tables.
4. Execute the flow and check the pandas task logs for the DataFrame preview.

How to extend

• Add a schedule or event trigger to run the extract automatically.
• Replace IonToCsv with a Parquet or JSON serializer for downstream tools.
• Expand the Python script to clean, aggregate, or write results elsewhere.
• Parameterize the SQL with inputs so the same flow serves multiple queries.

Links

• Databricks plugin documentation