DigitalOcean
DigitalOcean is the developer cloud, offering droplets, managed databases, and Kubernetes. With Kestra's dedicated plugin, teams orchestrate the whole account with event-driven flows.
Databricks
Databricks runs Spark, Delta Lake and SQL workloads on one data and AI platform. Kestra orchestrates the work around them: clusters, jobs, queries and the systems upstream and downstream.
About
The Databricks plugin covers the Databricks lifecycle from cluster creation to SQL queries, and Acxiom uses it to orchestrate Databricks job creation across isolated pipelines for its enterprise clients.
“Kestra was the only tool that combined true multi-tenant isolation, metadata-driven orchestration, and easy integration with our existing AWS and Databricks environments. It provided the foundation we needed to scale confidently.”
Databricks + Kestra: the lakehouse, orchestrated
Clusters that live only as long as the job
cluster.CreateCluster waits until the cluster is running and returns its clusterId. cluster.DeleteCluster removes it at the end of the flow, so no compute sits idle.
Jobs and one-off runs
job.CreateJob defines a Databricks job and job.SubmitRun runs a Spark or Python script on an existing cluster, then waits for the run to complete.
SQL from any flow
sql.Query runs SQL through a cluster's HTTP path and stores the result in Kestra internal storage with its row count. DatabricksCLI runs Databricks CLI commands.
Files in and out of DBFS
dbfs.Upload sends a file from Kestra internal storage to DBFS in 1 MB chunks, with no file size limit. A download task moves results back out.
