PLUGINSPLUGIN-DATABRICKSTASKSJOBIO.KESTRA.PLUGIN.DATABRICKS.JOB.SUBMITRUN

SubmitRun

SubmitRun

yaml

type: "io.kestra.plugin.databricks.job.SubmitRun"

Submit a Databricks run. Optionally, set waitForCompletion to a desired maximum duration to wait for the run completion.

Examples

Submit a Databricks run and wait up to 5 minutes for its completion

yaml

id: "submit_run"
type: "io.kestra.plugin.databricks.job.SubmitRun"
id: submitRun
type: io.kestra.plugin.databricks.job.SubmitRun
authentication:
  token: <your-token>
host: <your-host>
runTasks:
  - existingClusterId: <your-cluster>
    taskKey: taskKey
    sparkPythonTask:
      pythonFile: /Shared/hello.py
      sparkPythonTaskSource: WORKSPACE
waitForCompletion: PT5M

Properties

`runTasks`

Type: array
SubType: RunSubmitTaskSetting
Dynamic: ❌
Required: ✔️
Min items: 1

The run tasks, if multiple tasks are defined you must set dependsOn on each task

`accountId`

Type: string
Dynamic: ✔️
Required: ❌

Databricks account identifier

`authentication`

Type: AuthenticationConfig
Dynamic: ❌
Required: ❌

Databricks authentication configuration

This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.

`configFile`

Type: string
Dynamic: ✔️
Required: ❌

Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one

`host`

Type: string
Dynamic: ✔️
Required: ❌

Databricks host

`runName`

Type: string
Dynamic: ✔️
Required: ❌

The name of the run

`waitForCompletion`

Type: string
Dynamic: ❌
Required: ❌
Format: duration

If set, the task will wait for the run completion

Outputs

`runId`

Type: integer

The run identifier

`runURI`

Type: string

The run URI on the Databricks console

Definitions

`AuthenticationConfig`

`authType`

Type: string
Dynamic: ❌
Required: ❌

`azureClientId`

Type: string
Dynamic: ✔️
Required: ❌

`azureClientSecret`

Type: string
Dynamic: ✔️
Required: ❌

`azureTenantId`

Type: string
Dynamic: ✔️
Required: ❌

`clientId`

Type: string
Dynamic: ✔️
Required: ❌

`clientSecret`

Type: string
Dynamic: ✔️
Required: ❌

`googleCredentials`

Type: string
Dynamic: ✔️
Required: ❌

`googleServiceAccount`

Type: string
Dynamic: ✔️
Required: ❌

`password`

Type: string
Dynamic: ✔️
Required: ❌

`token`

Type: string
Dynamic: ✔️
Required: ❌

`username`

Type: string
Dynamic: ✔️
Required: ❌

`NotebookTaskSetting`

`baseParameters`

Type: object
SubType: string
Dynamic: ❌
Required: ❌

`notebookPath`

Type: string
Dynamic: ✔️
Required: ❌

`source`

Type: string
Dynamic: ❌
Required: ❌
Possible Values:
- GIT
- WORKSPACE

`SparkPythonTaskSetting`

`pythonFile`

Type: string
Dynamic: ✔️
Required: ✔️

`sparkPythonTaskSource`

Type: string
Dynamic: ❌
Required: ✔️
Possible Values:
- GIT
- WORKSPACE

`parameters`

Type: array
SubType: string
Dynamic: ✔️
Required: ❌

`PythonWheelTaskSetting`

`entryPoint`

Type: string
Dynamic: ✔️
Required: ❌

`namedParameters`

Type: object
SubType: string
Dynamic: ❌
Required: ❌

`packageName`

Type: string
Dynamic: ✔️
Required: ❌

`parameters`

Type: array
SubType: string
Dynamic: ✔️
Required: ❌

`PipelineTaskSetting`

`fullRefresh`

Type: boolean
Dynamic: ❌
Required: ❌

`pipelineId`

Type: string
Dynamic: ✔️
Required: ❌

`SparkJarTaskSetting`

`jarUri`

Type: string
Dynamic: ✔️
Required: ❌

`mainClassName`

Type: string
Dynamic: ✔️
Required: ❌

`parameters`

Type: array
SubType: string
Dynamic: ✔️
Required: ❌

`SparkSubmitTaskSetting`

`parameters`

Type: array
SubType: string
Dynamic: ✔️
Required: ❌

`RunSubmitTaskSetting`

`dependsOn`

Type: array
SubType: string
Dynamic: ❌
Required: ❌

Task dependencies, set this if multiple tasks are defined on the run

`existingClusterId`

Type: string
Dynamic: ✔️
Required: ❌

`notebookTask`

Type: NotebookTaskSetting
Dynamic: ❌
Required: ❌

Notebook task settings

`pipelineTask`

Type: PipelineTaskSetting
Dynamic: ❌
Required: ❌

Pipeline task settings

`pythonWheelTask`

Type: PythonWheelTaskSetting
Dynamic: ❌
Required: ❌

Python Wheel task settings

`sparkJarTask`

Type: SparkJarTaskSetting
Dynamic: ❌
Required: ❌

Spark JAR task settings

`sparkPythonTask`

Type: SparkPythonTaskSetting
Dynamic: ❌
Required: ❌

Spark Python task settings

`sparkSubmitTask`

Type: SparkSubmitTaskSetting
Dynamic: ❌
Required: ❌

Spark Submit task settings

`taskKey`

Type: string
Dynamic: ✔️
Required: ❌

`timeoutSeconds`

Type: integer
Dynamic: ❌
Required: ❌