SubmitRun
type: "io.kestra.plugin.databricks.job.SubmitRun"
Submit a Databricks run. Optionally, set
waitForCompletion
to a desired maximum duration to wait for the run completion.
Examples
Submit a Databricks run and wait up to 5 minutes for its completion
id: "submit_run"
type: "io.kestra.plugin.databricks.job.SubmitRun"
id: submitRun
type: io.kestra.plugin.databricks.job.SubmitRun
authentication:
token: <your-token>
host: <your-host>
runTasks:
- existingClusterId: <your-cluster>
taskKey: taskKey
sparkPythonTask:
pythonFile: /Shared/hello.py
sparkPythonTaskSource: WORKSPACE
waitForCompletion: PT5M
Properties
runTasks
- Type: array
- SubType: RunSubmitTaskSetting
- Dynamic: ❌
- Required: ✔️
- Min items:
1
The run tasks, if multiple tasks are defined you must set
dependsOn
on each task
accountId
- Type: string
- Dynamic: ✔️
- Required: ❌
Databricks account identifier
authentication
- Type: AuthenticationConfig
- Dynamic: ❌
- Required: ❌
Databricks authentication configuration
This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.
configFile
- Type: string
- Dynamic: ✔️
- Required: ❌
Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one
host
- Type: string
- Dynamic: ✔️
- Required: ❌
Databricks host
runName
- Type: string
- Dynamic: ✔️
- Required: ❌
The name of the run
waitForCompletion
- Type: string
- Dynamic: ❌
- Required: ❌
- Format:
duration
If set, the task will wait for the run completion
Outputs
runId
- Type: integer
The run identifier
runURI
- Type: string
The run URI on the Databricks console
Definitions
AuthenticationConfig
authType
- Type: string
- Dynamic: ❌
- Required: ❌
azureClientId
- Type: string
- Dynamic: ✔️
- Required: ❌
azureClientSecret
- Type: string
- Dynamic: ✔️
- Required: ❌
azureTenantId
- Type: string
- Dynamic: ✔️
- Required: ❌
clientId
- Type: string
- Dynamic: ✔️
- Required: ❌
clientSecret
- Type: string
- Dynamic: ✔️
- Required: ❌
googleCredentials
- Type: string
- Dynamic: ✔️
- Required: ❌
googleServiceAccount
- Type: string
- Dynamic: ✔️
- Required: ❌
password
- Type: string
- Dynamic: ✔️
- Required: ❌
token
- Type: string
- Dynamic: ✔️
- Required: ❌
username
- Type: string
- Dynamic: ✔️
- Required: ❌
NotebookTaskSetting
baseParameters
- Type: object
- SubType: string
- Dynamic: ❌
- Required: ❌
notebookPath
- Type: string
- Dynamic: ✔️
- Required: ❌
source
- Type: string
- Dynamic: ❌
- Required: ❌
- Possible Values:
GIT
WORKSPACE
SparkPythonTaskSetting
pythonFile
- Type: string
- Dynamic: ✔️
- Required: ✔️
sparkPythonTaskSource
- Type: string
- Dynamic: ❌
- Required: ✔️
- Possible Values:
GIT
WORKSPACE
parameters
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
PythonWheelTaskSetting
entryPoint
- Type: string
- Dynamic: ✔️
- Required: ❌
namedParameters
- Type: object
- SubType: string
- Dynamic: ❌
- Required: ❌
packageName
- Type: string
- Dynamic: ✔️
- Required: ❌
parameters
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
PipelineTaskSetting
fullRefresh
- Type: boolean
- Dynamic: ❌
- Required: ❌
pipelineId
- Type: string
- Dynamic: ✔️
- Required: ❌
SparkJarTaskSetting
jarUri
- Type: string
- Dynamic: ✔️
- Required: ❌
mainClassName
- Type: string
- Dynamic: ✔️
- Required: ❌
parameters
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
SparkSubmitTaskSetting
parameters
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
RunSubmitTaskSetting
dependsOn
- Type: array
- SubType: string
- Dynamic: ❌
- Required: ❌
Task dependencies, set this if multiple tasks are defined on the run
existingClusterId
- Type: string
- Dynamic: ✔️
- Required: ❌
notebookTask
- Type: NotebookTaskSetting
- Dynamic: ❌
- Required: ❌
Notebook task settings
pipelineTask
- Type: PipelineTaskSetting
- Dynamic: ❌
- Required: ❌
Pipeline task settings
pythonWheelTask
- Type: PythonWheelTaskSetting
- Dynamic: ❌
- Required: ❌
Python Wheel task settings
sparkJarTask
- Type: SparkJarTaskSetting
- Dynamic: ❌
- Required: ❌
Spark JAR task settings
sparkPythonTask
- Type: SparkPythonTaskSetting
- Dynamic: ❌
- Required: ❌
Spark Python task settings
sparkSubmitTask
- Type: SparkSubmitTaskSetting
- Dynamic: ❌
- Required: ❌
Spark Submit task settings
taskKey
- Type: string
- Dynamic: ✔️
- Required: ❌
timeoutSeconds
- Type: integer
- Dynamic: ❌
- Required: ❌