Databricks Workflows is the job orchestration layer built into the Databricks lakehouse. Kestra is open-source workflow orchestration for any cloud, any language, and use cases beyond data transformation. One is built to coordinate jobs inside Databricks. The other orchestrates everything your engineering team ships.
Declarative YAML workflows versioned in Git, executed in isolated containers, deployed through CI/CD. Orchestrate data pipelines, infrastructure operations, AI workloads, and business processes across AWS, Azure, GCP, or on-premises without vendor lock-in.
Multi-task job scheduling built into the Databricks workspace. Chain notebooks, Python scripts, JARs, SQL tasks, and Delta Live Tables pipelines with dependency management and retry logic. Pricing runs on Databricks job compute clusters.
Databricks Workflows runs on Databricks' cloud platform (AWS, Azure, or GCP)—there is no local install option. This comparison reflects what's required to provision a workspace, configure compute, and define your first job.
curl -o docker-compose.yml \https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.ymldocker compose up
# Open localhost:8080# Pick a Blueprint, run it. Done.Download the Docker Compose file, spin it up, and you're ready. Database and config included. Open the UI, pick a Blueprint, run it. No cloud account, no cluster provisioning, no workspace setup.
# 1. Provision Databricks workspace (AWS, Azure, or GCP)# 2. Configure cloud IAM and storage permissions# 3. Create or select a job cluster (instance type, DBR version)# 4. Upload notebooks or Python scripts to DBFS or repos# 5. Define job tasks and dependencies in the UI or via API:
curl -X POST https://<workspace>/api/2.1/jobs/create \ -H "Authorization: Bearer $DATABRICKS_TOKEN" \ -d '{ "name": "daily_etl", "tasks": [{ "task_key": "ingest", "notebook_task": {...} }] }'Requires a Databricks workspace, a cloud account with compute permissions, cluster configuration, and either navigating the Jobs UI or writing JSON job definitions via the REST API. Teams also need to package notebooks and scripts into the Databricks environment before scheduling them.
YAML is readable on day 1. Our docs are embedded in the UI for easy reference, the AI Copilot writes workflows for you, or start with our library of Blueprints. Every workflow is a file in your repository, reviewed in pull requests, deployed the same way as application code.
Workflows are defined in the Databricks UI or as JSON via the Jobs REST API. Version control requires manually committing JSON exports or using the Databricks Repos integration. The JSON schema is verbose and Databricks-specific, with cluster configuration embedded in every job definition.
Orchestrate data pipelines, infrastructure operations, AI workloads, and business processes across any cloud or on-premises environment. Event-driven at its core, with native triggers for S3, webhooks, Kafka, database changes, and API events. Run Databricks jobs as one step in a broader workflow.
Jobs follow a task graph model: define tasks (notebooks, scripts, SQL, DLT pipelines), set dependencies, attach compute, and schedule or trigger by file arrival. All execution happens on Databricks clusters. Monitoring and logs are in the Databricks workspace UI.
| | | |
|---|---|---|
| Deployment model | Self-hosted (Docker, Kubernetes) or Kestra Cloud | Databricks-managed (requires Databricks workspace) |
| Workflow definition | Declarative YAML | UI builder or JSON via REST API |
| Version control | Native Git and CI/CD | Requires Databricks Repos or manual JSON export |
| Cloud support | Multi-cloud and on-premises | Databricks-only (AWS, Azure, or GCP via Databricks) |
| Languages supported | Any (Python, SQL, Bash, Go, R, Node.js) | Python, Scala, SQL, notebooks (Databricks runtime) |
| Open source | Apache 2.0 | No open-source version |
| Infrastructure automation | Native support | Not designed for this |
| Self-service for non-engineers | Kestra Apps | Monitoring UI only |
| Pricing model | Free open-source core (Enterprise tier available) | Databricks DBU consumption on job compute clusters |
| Air-gapped deployment | Supported | Not available (requires Databricks cloud) |
| Multi-tenancy | Namespace isolation + RBAC out-of-box | Workspace-level isolation with Databricks RBAC |
Kestra was the only tool that combined true multi-tenant isolation, metadata-driven orchestration, and easy integration with our existing AWS and Databricks environments. It provided the foundation we needed to scale confidently.
Kestra handles the full pipeline from data ingestion through infrastructure changes and downstream notifications. Trigger Terraform runs after a data load, coordinate cross-team approvals, run dbt on any warehouse, and notify downstream services — all in one YAML file. Every system your team operates becomes one step in a unified workflow.
Kestra workflows are YAML files from day one: they live in your repo, go through code review, and deploy through CI/CD the same way as application code. Git sync, a Terraform provider, and native CI/CD hooks mean workflow deployment follows the same process as every other piece of infrastructure your team ships.
Kestra runs on any infrastructure: Docker, Kubernetes, AWS, Azure, GCP, or on-premises. Trigger Spark jobs when you need distributed compute, and run everything else on your own infrastructure without per-execution overhead. One orchestrator connects every system regardless of where it runs.
Find answers to your questions right here, and don't hesitate to Contact Us if you couldn't find what you're looking for.
See how Kestra can simplify your data workflows—and orchestrate across the full stack, not just Databricks.