SubmitRun
Submit a Databricks run. Optionally, set waitForCompletion to a desired maximum duration to wait for the run completion.
type: "io.kestra.plugin.databricks.job.SubmitRun"Submit a Databricks run and wait up to 5 minutes for its completion.
id: databricks_job_submit_run
namespace: company.team
tasks:
- id: submit_run
type: io.kestra.plugin.databricks.job.SubmitRun
authentication:
token: <your-token>
host: <your-host>
runTasks:
- existingClusterId: <your-cluster>
taskKey: taskKey
sparkPythonTask:
pythonFile: /Shared/hello.py
sparkPythonTaskSource: WORKSPACE
waitForCompletion: PT5M
YESDatabricks account identifier.
YESDatabricks authentication configuration.
This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.
YESDatabricks configuration file, use this if you don't want to configure each Databricks account properties one by one.
YESDatabricks host.
YESThe name of the run.
YESdurationIf set, the task will wait for the run completion.
The run identifier.
uriThe run URI on the Databricks console.
YESMap of task base parameters.
Can be a map of string/string or a variable that binds to a JSON object.
YESYESGITWORKSPACEYESYESMap of task named parameters.
Can be a map of string/string or a variable that binds to a JSON object.
YESYESList of task parameters.
Can be a list of strings or a variable that binds to a JSON array of strings.
YESYESYESList of task parameters.
Can be a list of strings or a variable that binds to a JSON array of strings.
NOTask dependencies, set this if multiple tasks are defined on the run.
YESNONotebook task settings.
NOPipeline task settings.
NOPython Wheel task settings.
NORun job task settings.
NOSpark JAR task settings.
NOSpark Python task settings.
NOSpark Submit task settings.
YESNOYESYESYESYESYESYESYESYESYESYESYESYESYESYESYESGITWORKSPACEYESList of task parameters.
Can be a list of strings or a variable that binds to a JSON array of strings.
YESYESYESYESYESYESYESYESYESYESYESYESYESList of task parameters.
Can be a list of strings or a variable that binds to a JSON array of strings.
YESYESYES