Submit a Databricks run.
Optionally, set waitForCompletion
to a desired maximum duration to wait for the run completion.
type: "io.kestra.plugin.databricks.job.submitrun"
Examples
Submit a Databricks run and wait up to 5 minutes for its completion.
id: databricks_job_submit_run
namespace: company.team
tasks:
- id: submit_run
type: io.kestra.plugin.databricks.job.SubmitRun
host: "{{ secret('DATABRICKS_HOST') }}"
authentication:
token: "{{ secret('DATABRICKS_TOKEN') }}"
runTasks:
- existingClusterId: <your-cluster>
taskKey: pysparkTask
sparkPythonTask:
pythonFile: /Shared/hello.py
sparkPythonTaskSource: WORKSPACE
waitForCompletion: PT5M
Properties
runTasks *RequiredNon-dynamicarray
accountId string
Databricks account identifier.
authentication AbstractTask-AuthenticationConfig
Databricks authentication configuration.
This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.
configFile string
Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one.
host string
Databricks host.
runName string
The name of the run.
waitForCompletion string
duration
If set, the task will wait for the run completion.
Outputs
runId integer
The run identifier.
runURI string
uri
The run URI on the Databricks console.
Definitions
io.kestra.plugin.databricks.job.task.NotebookTaskSetting
baseParameters stringobject
Map of task base parameters.
Can be a map of string/string or a variable that binds to a JSON object.
notebookPath string
source string
GIT
WORKSPACE
io.kestra.plugin.databricks.job.task.PythonWheelTaskSetting
entryPoint string
namedParameters stringobject
Map of task named parameters.
Can be a map of string/string or a variable that binds to a JSON object.
packageName string
parameters stringarray
List of task parameters.
Can be a list of strings or a variable that binds to a JSON array of strings.
io.kestra.plugin.databricks.job.task.LibrarySetting-CranSetting
_package string
repo string
io.kestra.plugin.databricks.job.task.SparkSubmitTaskSetting
parameters stringarray
List of task parameters.
Can be a list of strings or a variable that binds to a JSON array of strings.
io.kestra.plugin.databricks.job.SubmitRun-RunSubmitTaskSetting
dependsOn array
Task dependencies, set this if multiple tasks are defined on the run.
existingClusterId string
notebookTask NotebookTaskSetting
Notebook task settings.
pipelineTask PipelineTaskSetting
Pipeline task settings.
pythonWheelTask PythonWheelTaskSetting
Python Wheel task settings.
runJobTask RunJobTaskSetting
Run job task settings.
sparkJarTask SparkJarTaskSetting
Spark JAR task settings.
sparkPythonTask SparkPythonTaskSetting
Spark Python task settings.
sparkSubmitTask SparkSubmitTaskSetting
Spark Submit task settings.
taskKey string
timeoutSeconds integer
io.kestra.plugin.databricks.AbstractTask-AuthenticationConfig
authType string
azureClientId string
azureClientSecret string
azureTenantId string
clientId string
clientSecret string
googleCredentials string
googleServiceAccount string
password string
token string
username string
io.kestra.plugin.databricks.job.task.RunJobTaskSetting
jobId string
jobParameters object
io.kestra.plugin.databricks.job.task.SparkPythonTaskSetting
pythonFile *Requiredstring
sparkPythonTaskSource *Requiredstring
GIT
WORKSPACE
parameters stringarray
List of task parameters.
Can be a list of strings or a variable that binds to a JSON array of strings.
io.kestra.plugin.databricks.job.task.PipelineTaskSetting
fullRefresh booleanstring
pipelineId string
io.kestra.plugin.databricks.job.task.LibrarySetting
egg string
jar string
whl string
io.kestra.plugin.databricks.job.task.LibrarySetting-PypiSetting
_package string
repo string
io.kestra.plugin.databricks.job.task.SparkJarTaskSetting
jarUri string
mainClassName string
parameters stringarray
List of task parameters.
Can be a list of strings or a variable that binds to a JSON array of strings.