Run a task on an on-demand Databricks cluster

Run ephemeral Spark workloads on Databricks without leaving idle clusters running and burning cloud spend. This blueprint provisions a fresh Databricks cluster on demand, executes a Spark Python job on it, and deletes the cluster afterwards, so you pay only for the compute you actually use. The cleanup is guaranteed even when the job fails, which makes it a reliable pattern for scheduled batch jobs, ad-hoc data processing, and CI-style pipeline runs on Databricks.

How it works

1. The create_cluster task (io.kestra.plugin.databricks.cluster.CreateCluster) provisions a new cluster named kestra-demo with the requested nodeTypeId, numWorkers, and sparkVersion, authenticating against your workspace host with a token.
2. The run_job task (io.kestra.plugin.databricks.job.CreateJob) submits a Spark Python job to the cluster created above, referencing it through {{ outputs.create_cluster.clusterId }}. It runs the pythonFile stored in the Databricks workspace and waits up to five minutes for completion, as set on waitForCompletion.
3. The job runs inside an io.kestra.plugin.core.flow.AllowFailure block, so a failed job does not abort the flow.
4. The delete_cluster task (io.kestra.plugin.databricks.cluster.DeleteCluster) always runs and removes the cluster, ensuring no compute is left running regardless of the job outcome.

What you get

• On-demand compute that exists only for the duration of the job.
• Guaranteed cluster teardown even when the Spark job fails.
• A clear, declarative definition of provisioning, execution, and cleanup in one file.
• Outputs (such as the cluster id) wired automatically between tasks.

Who it's for

• Data engineers running scheduled or ad-hoc Spark batch jobs on Databricks.
• Platform teams that want to cap cloud spend by avoiding always-on clusters.
• ML and analytics teams orchestrating Databricks steps as part of larger pipelines.

Why orchestrate this with Kestra

Databricks job scheduling assumes you manage cluster lifecycle yourself, and a job that fails can leave a cluster running. Kestra makes the full lifecycle declarative: provision, run, and delete in one versioned YAML file, with the AllowFailure block guaranteeing cleanup. You can drive runs from event triggers, schedules, or upstream tasks, add retries, and capture lineage across the whole flow, not just inside a single Databricks job.

Prerequisites

• A Databricks workspace with permission to create and delete clusters and submit jobs.
• The Spark Python file (for example /Shared/hello.py) available in the workspace.

Secrets

• DATABRICKS_HOST: your Databricks workspace URL.
• DATABRICKS_TOKEN: a personal access token used to authenticate API calls.

Quick start

1. Add the DATABRICKS_HOST and DATABRICKS_TOKEN secrets to your Kestra instance.
2. Upload your Spark Python file to the Databricks workspace and update pythonFile to match.
3. Adjust nodeTypeId, numWorkers, and sparkVersion to fit your workload.
4. Add the flow and trigger an execution.

How to extend

• Add a Schedule or event trigger to run the job on a cadence or on incoming data.
• Replace sparkPythonTask with a notebook, JAR, or SQL task in jobTasks.
• Add retries on run_job and notifications on failure.
• Pass parameters into the job via inputs and templated values.

Links

• Databricks plugin docs