Databricks runs the compute. Kestra decides when a run starts, what feeds it, what checks it, and what runs next. Submit Databricks runs from a YAML flow, wait for them, gate on SQL checks, and chain them with ingestion, dbt and alerting in one execution history.
In short
Kestra orchestrates Databricks from outside the workspace. A YAML flow submits a Databricks run with the SubmitRun task, waits for it to finish, runs SQL checks on a SQL warehouse, and chains the result with ingestion, dbt and alerting. Flows start on a schedule, a file arrival, a new Unity Catalog table or an API call.
Updated · Kestra 2.0
databricks.job.SubmitRun submits the run, waits for it, and adopts it after a worker lossDatabricks Lakeflow Jobs schedule work inside the workspace. Kestra covers everything around it: it starts Databricks runs when upstream data is ready, runs SQL quality gates before anyone reads the result, creates and deletes clusters per run, and chains Databricks with Airbyte ingestion, dbt transforms and Slack alerts in one flow. Databricks keeps doing the compute; Kestra gives one execution history across every tool.
Databricks runs the notebooks, the Spark code and the SQL warehouse. Kestra runs the steps around them: what triggers a run, what it depends on, which checks it must pass, and where the cross-tool audit trail lives.
SubmitRun sends one or more tasks (notebook, Spark Python, Spark JAR, Spark Submit, Python wheel or pipeline) as a Databricks run and waits for a terminal state. The submission is idempotent: if the Kestra worker is lost mid-run, the resubmitted task adopts the run already in flight instead of launching a second one.
Start a flow when a new table appears in a Unity Catalog schema with the unitycatalog.table.Trigger, when a Lakebase query returns rows, or on events Databricks does not see: an S3 file, a Kafka message, a webhook, or another flow finishing.
Ingestion with Airbyte, the Databricks run, a dbt build and a notification run as ordered steps of one flow. A failed dbt step retries on its own; the Databricks run does not start again.
CreateCluster provisions a cluster, the job runs on it, and DeleteCluster removes it in the same flow, so compute exists only for the job. A scheduled audit flow can also find clusters left running outside working hours.
sql.Query runs checks on a cluster or SQL warehouse and returns the row count. A Fail or If task turns that result into a gate, so a bad load stops the pipeline instead of reaching dashboards.
Create and update catalogs, schemas, volumes and grants from a flow, read table metadata, and run compliance checks on permissions on a schedule. Every change lands in the execution history with the run that made it.
Four steps take a Databricks job from a manual run to a scheduled, checked and alerted pipeline. The flow below puts them together.
Save the workspace host and a personal access token (or an OAuth M2M service principal) as Kestra secrets, so no credential appears in the flow YAML.
Use io.kestra.plugin.databricks.job.SubmitRun with an existing cluster or a cluster created in the same flow. waitForCompletion keeps the task running until Databricks reports a terminal state, and the run ID and console URL become task outputs.
Run a check with io.kestra.plugin.databricks.sql.Query on a SQL warehouse. If the query returns bad rows, a Fail task stops the flow before anything downstream reads the table.
Add a Schedule trigger, or start the flow on an event such as a new Unity Catalog table, an S3 file or a webhook. An errors branch posts to Slack with the execution ID when any step fails.
Patterns data platform teams run in production. Each one shows the flow end to end, with the real plugin classes in play.
Airbyte loads raw data, Kestra submits the Databricks transformation and waits for it, then runs dbt on the curated tables. Slack confirms or alerts. Each step retries on its own.
The Databricks run ID and console URL are task outputs available to every later step.
A dbt failure retries dbt only; the Databricks run is not resubmitted.
Ingestion, compute and transformation share one execution ID and one log.
Create a cluster sized for the job, run the job on it, and delete the cluster when the job ends, even if the job fails, so no cluster is left running between runs.
The job runs on the cluster the previous task created.
The delete step runs even when the job fails.
A scheduled flow lists clusters still running outside working hours.
The Unity Catalog table trigger polls a schema and starts the flow when a table appears. The flow reads its metadata, runs a validation query and posts the result.
The trigger stores the tables it has already seen in the KV Store.
React to new tables, updated tables, or both.
Column definitions come from the Get task, ready for checks.
| Capability | Apache Airflow | Dagster | ||
|---|---|---|---|---|
| Submit a Databricks run and wait for it | SubmitRun / CreateJob tasks | Native | Databricks provider operators | dagster-databricks integration |
| Steps outside Databricks (other clouds, on-prem, SaaS) | Native plugins in the same flow | Limited to workspace tasks and API calls | Provider operators | Python resources |
| Event triggers outside Databricks (S3, Kafka, webhooks) | Native triggers | File arrival and table update triggers on Databricks storage | Sensors and datasets | Sensors, Python required |
| Workflow definition | Declarative YAML | UI, JSON or Asset Bundles | Python DAGs | Python assets |
| Infrastructure and business steps in the same pipeline | Terraform, ServiceNow and 2,600+ plugins | No | Provider-dependent | Python code |
| Runs without a Databricks dependency | Self-hosted or cloud | Databricks only | Self-hosted or managed | Self-hosted or managed |
Find answers to your questions right here, and don't hesitate to Contact Us if you couldn't find what you're looking for.
Databricks Workflows alternatives
When teams look beyond Databricks Workflows, and how the main orchestrators compare.
Databricks vs Snowflake
How the two platforms differ for data engineering, analytics and AI workloads.
Simplifying Databricks Workflow Management with Kestra
Running and coordinating Databricks jobs from Kestra alongside the rest of the data stack.
Databricks plugin reference
Every Databricks task and trigger with its properties.
Submit runs, gate on data quality, use ephemeral clusters, and chain Databricks with ingestion and dbt in one flow. Open source, self-hosted, event-driven.