🚀 New! Kestra raises $3 million to grow Learn more

SubmitRun SubmitRun

yaml
type: "io.kestra.plugin.databricks.job.SubmitRun"

Submit a Databricks run. Optionally, set waitForCompletion to a desired maximum duration to wait for the run completion.

Examples

Submit a Databricks run and wait up to 5 minutes for its completion

yaml
id: "submit_run"
type: "io.kestra.plugin.databricks.job.SubmitRun"
id: submitRun
type: io.kestra.plugin.databricks.job.SubmitRun
authentication:
  token: <your-token>
host: <your-host>
runTasks:
  - existingClusterId: <your-cluster>
    taskKey: taskKey
    sparkPythonTask:
      pythonFile: /Shared/hello.py
      sparkPythonTaskSource: WORKSPACE
waitForCompletion: PT5M

Properties

runTasks

The run tasks, if multiple tasks are defined you must set dependsOn on each task

accountId

  • Type: string
  • Dynamic: ✔️
  • Required:

Databricks account identifier

authentication

Databricks authentication configuration This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.

configFile

  • Type: string
  • Dynamic: ✔️
  • Required:

Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one

host

  • Type: string
  • Dynamic: ✔️
  • Required:

Databricks host

runName

  • Type: string
  • Dynamic: ✔️
  • Required:

The name of the run

waitForCompletion

  • Type: string
  • Dynamic:
  • Required:
  • Format: duration

If set, the task will wait for the run completion

Outputs

runId

  • Type: integer
  • Dynamic:
  • Required:

The run identifier

runURI

  • Type: string
  • Dynamic:
  • Required:
  • Format: uri

The run URI on the Databricks console

Definitions

io.kestra.plugin.databricks.AbstractTask-AuthenticationConfig

Properties

authType
  • Type: string
  • Dynamic: ✔️
  • Required:
azureClientId
  • Type: string
  • Dynamic: ✔️
  • Required:
azureClientSecret
  • Type: string
  • Dynamic: ✔️
  • Required:
azureTenantId
  • Type: string
  • Dynamic: ✔️
  • Required:
clientId
  • Type: string
  • Dynamic: ✔️
  • Required:
clientSecret
  • Type: string
  • Dynamic: ✔️
  • Required:
googleCredentials
  • Type: string
  • Dynamic: ✔️
  • Required:
googleServiceAccount
  • Type: string
  • Dynamic: ✔️
  • Required:
password
  • Type: string
  • Dynamic: ✔️
  • Required:
token
  • Type: string
  • Dynamic: ✔️
  • Required:
username
  • Type: string
  • Dynamic: ✔️
  • Required:

io.kestra.plugin.databricks.job.task.NotebookTaskSetting

Properties

baseParameters
  • Type:
    • string
    • object
  • SubType: string
  • Dynamic: ✔️
  • Required:

Map of task base parameters. Can be a map of string/string or a variable that binds to a JSON object.

notebookPath
  • Type: string
  • Dynamic: ✔️
  • Required:
source
  • Type: string
  • Dynamic:
  • Required:
  • Possible Values:
    • GIT
    • WORKSPACE

io.kestra.plugin.databricks.job.task.SparkPythonTaskSetting

Properties

pythonFile
  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
sparkPythonTaskSource
  • Type: string
  • Dynamic:
  • Required: ✔️
  • Possible Values:
    • GIT
    • WORKSPACE
parameters
  • Type:
    • string
    • array
  • Dynamic: ✔️
  • Required:

List of task parameters. Can be a list of strings or a variable that binds to a JSON array of strings.

io.kestra.plugin.databricks.job.task.PythonWheelTaskSetting

Properties

entryPoint
  • Type: string
  • Dynamic: ✔️
  • Required:
namedParameters
  • Type:
    • string
    • object
  • SubType: string
  • Dynamic: ✔️
  • Required:

Map of task named parameters. Can be a map of string/string or a variable that binds to a JSON object.

packageName
  • Type: string
  • Dynamic: ✔️
  • Required:
parameters
  • Type:
    • string
    • array
  • Dynamic: ✔️
  • Required:

List of task parameters. Can be a list of strings or a variable that binds to a JSON array of strings.

io.kestra.plugin.databricks.job.task.PipelineTaskSetting

Properties

fullRefresh
  • Type: boolean
  • Dynamic:
  • Required:
pipelineId
  • Type: string
  • Dynamic: ✔️
  • Required:

io.kestra.plugin.databricks.job.task.LibrarySetting

Properties

cran
egg
  • Type: string
  • Dynamic: ✔️
  • Required:
jar
  • Type: string
  • Dynamic: ✔️
  • Required:
maven
pypi
whl
  • Type: string
  • Dynamic: ✔️
  • Required:

io.kestra.plugin.databricks.job.task.LibrarySetting-CranSetting

Properties

_package
  • Type: string
  • Dynamic: ✔️
  • Required:
repo
  • Type: string
  • Dynamic: ✔️
  • Required:

io.kestra.plugin.databricks.job.task.LibrarySetting-PypiSetting

Properties

_package
  • Type: string
  • Dynamic: ✔️
  • Required:
repo
  • Type: string
  • Dynamic: ✔️
  • Required:

io.kestra.plugin.databricks.job.task.SparkJarTaskSetting

Properties

jarUri
  • Type: string
  • Dynamic: ✔️
  • Required:
mainClassName
  • Type: string
  • Dynamic: ✔️
  • Required:
parameters
  • Type:
    • string
    • array
  • Dynamic: ✔️
  • Required:

List of task parameters. Can be a list of strings or a variable that binds to a JSON array of strings.

io.kestra.plugin.databricks.job.task.SparkSubmitTaskSetting

Properties

parameters
  • Type:
    • string
    • array
  • Dynamic: ✔️
  • Required:

List of task parameters. Can be a list of strings or a variable that binds to a JSON array of strings.

io.kestra.plugin.databricks.job.SubmitRun-RunSubmitTaskSetting

Properties

dependsOn
  • Type: array
  • SubType: string
  • Dynamic:
  • Required:

Task dependencies, set this if multiple tasks are defined on the run

existingClusterId
  • Type: string
  • Dynamic: ✔️
  • Required:
libraries

Task libraries

notebookTask

Notebook task settings

pipelineTask

Pipeline task settings

pythonWheelTask

Python Wheel task settings

sparkJarTask

Spark JAR task settings

sparkPythonTask

Spark Python task settings

sparkSubmitTask

Spark Submit task settings

taskKey
  • Type: string
  • Dynamic: ✔️
  • Required:
timeoutSeconds
  • Type: integer
  • Dynamic:
  • Required:

io.kestra.plugin.databricks.job.task.LibrarySetting-MavenSetting

Properties

coordinates
  • Type: string
  • Dynamic: ✔️
  • Required:
exclusions
  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required:
repo
  • Type: string
  • Dynamic: ✔️
  • Required: