SubmitRun SubmitRun

yaml
type: "io.kestra.plugin.databricks.job.SubmitRun"

Submit a Databricks run. Optionally, set waitForCompletion to a desired maximum duration to wait for the run completion.

Examples

Submit a Databricks run and wait up to 5 minutes for its completion

yaml
id: "submit_run"
type: "io.kestra.plugin.databricks.job.SubmitRun"
id: submitRun
type: io.kestra.plugin.databricks.job.SubmitRun
authentication:
  token: <your-token>
host: <your-host>
runTasks:
  - existingClusterId: <your-cluster>
    taskKey: taskKey
    sparkPythonTask:
      pythonFile: /Shared/hello.py
      sparkPythonTaskSource: WORKSPACE
waitForCompletion: PT5M

Properties

runTasks

The run tasks, if multiple tasks are defined you must set dependsOn on each task

accountId

  • Type: string
  • Dynamic: ✔️
  • Required:

Databricks account identifier

authentication

Databricks authentication configuration

This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.

configFile

  • Type: string
  • Dynamic: ✔️
  • Required:

Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one

host

  • Type: string
  • Dynamic: ✔️
  • Required:

Databricks host

runName

  • Type: string
  • Dynamic: ✔️
  • Required:

The name of the run

waitForCompletion

  • Type: string
  • Dynamic:
  • Required:
  • Format: duration

If set, the task will wait for the run completion

Outputs

runId

  • Type: integer

The run identifier

runURI

  • Type: string

The run URI on the Databricks console

Definitions

AuthenticationConfig

authType

  • Type: string
  • Dynamic:
  • Required:

azureClientId

  • Type: string
  • Dynamic: ✔️
  • Required:

azureClientSecret

  • Type: string
  • Dynamic: ✔️
  • Required:

azureTenantId

  • Type: string
  • Dynamic: ✔️
  • Required:

clientId

  • Type: string
  • Dynamic: ✔️
  • Required:

clientSecret

  • Type: string
  • Dynamic: ✔️
  • Required:

googleCredentials

  • Type: string
  • Dynamic: ✔️
  • Required:

googleServiceAccount

  • Type: string
  • Dynamic: ✔️
  • Required:

password

  • Type: string
  • Dynamic: ✔️
  • Required:

token

  • Type: string
  • Dynamic: ✔️
  • Required:

username

  • Type: string
  • Dynamic: ✔️
  • Required:

NotebookTaskSetting

baseParameters

  • Type: object
  • SubType: string
  • Dynamic:
  • Required:

notebookPath

  • Type: string
  • Dynamic: ✔️
  • Required:

source

  • Type: string
  • Dynamic:
  • Required:
  • Possible Values:
    • GIT
    • WORKSPACE

SparkPythonTaskSetting

pythonFile

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

sparkPythonTaskSource

  • Type: string
  • Dynamic:
  • Required: ✔️
  • Possible Values:
    • GIT
    • WORKSPACE

parameters

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required:

PythonWheelTaskSetting

entryPoint

  • Type: string
  • Dynamic: ✔️
  • Required:

namedParameters

  • Type: object
  • SubType: string
  • Dynamic:
  • Required:

packageName

  • Type: string
  • Dynamic: ✔️
  • Required:

parameters

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required:

PipelineTaskSetting

fullRefresh

  • Type: boolean
  • Dynamic:
  • Required:

pipelineId

  • Type: string
  • Dynamic: ✔️
  • Required:

SparkJarTaskSetting

jarUri

  • Type: string
  • Dynamic: ✔️
  • Required:

mainClassName

  • Type: string
  • Dynamic: ✔️
  • Required:

parameters

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required:

SparkSubmitTaskSetting

parameters

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required:

RunSubmitTaskSetting

dependsOn

  • Type: array
  • SubType: string
  • Dynamic:
  • Required:

Task dependencies, set this if multiple tasks are defined on the run

existingClusterId

  • Type: string
  • Dynamic: ✔️
  • Required:

notebookTask

Notebook task settings

pipelineTask

Pipeline task settings

pythonWheelTask

Python Wheel task settings

sparkJarTask

Spark JAR task settings

sparkPythonTask

Spark Python task settings

sparkSubmitTask

Spark Submit task settings

taskKey

  • Type: string
  • Dynamic: ✔️
  • Required:

timeoutSeconds

  • Type: integer
  • Dynamic:
  • Required: