CreateJob
Create a Databricks job and run it. Set waitForCompletion to the desired maximum duration if you want the task to wait for the job completion (e.g., PT1H to wait up to one hour).
type: "io.kestra.plugin.databricks.job.CreateJob"Create a Databricks job, run it, and wait for completion for five minutes.
id: databricks_job_create
namespace: company.team
tasks:
- id: create_job
type: io.kestra.plugin.databricks.job.CreateJob
authentication:
token: <your-token>
host: <your-host>
jobTasks:
- existingClusterId: <your-cluster>
taskKey: taskKey
sparkPythonTask:
pythonFile: /Shared/hello.py
sparkPythonTaskSource: WORKSPACE
waitForCompletion: PT5M
YESDatabricks account identifier.
YESDatabricks authentication configuration.
This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.
YESDatabricks configuration file, use this if you don't want to configure each Databricks account properties one by one.
YESDatabricks host.
YESThe name of the job.
YESdurationIf set, the task will wait for the job run completion for up to the waitForCompletion duration before timing out.
The job identifier.
uriThe job URI on the Databricks console.
The run identifier.
uriThe run URI on the Databricks console.
YESMap of task parameters.
Can be a map of string/string or a variable that binds to a JSON object.
YESYESYESMap of task base parameters.
Can be a map of string/string or a variable that binds to a JSON object.
YESYESGITWORKSPACEYESDBT task settings.
YESTask dependencies, set this if multiple tasks are defined on the job.
YESTask description.
YESThe identifier of the cluster.
YESNotebook task settings.
YESPipeline task settings.
YESPython Wheel task settings.
YESRun job task settings.
YESSpark JAR task settings.
YESSpark Python task settings.
YESSpark Submit task settings.
YESSQL task settings.
YESTask key.
YESYESYESMap of task named parameters.
Can be a map of string/string or a variable that binds to a JSON object.
YESYESList of task parameters.
Can be a list of strings or a variable that binds to a JSON array of strings.
YESYESYESList of task parameters.
Can be a list of strings or a variable that binds to a JSON array of strings.
YESYESYESYESYESYESYESYESYESYESYESYESYESYESYESGITWORKSPACEYESList of task parameters.
Can be a list of strings or a variable that binds to a JSON array of strings.
YESYESYESYESYESYESYESYESYESYESYESYESYESYESYESYESYESList of task parameters.
Can be a list of strings or a variable that binds to a JSON array of strings.
YESYESYES