Submit a Databricks run.

Optionally, set waitForCompletion to a desired maximum duration to wait for the run completion.

yaml
type: "io.kestra.plugin.databricks.job.submitrun"

Examples

Submit a Databricks run and wait up to 5 minutes for its completion.

yaml
id: databricks_job_submit_run
namespace: company.team

tasks:
  - id: submit_run
    type: io.kestra.plugin.databricks.job.SubmitRun
    host: "{{ secret('DATABRICKS_HOST') }}"
    authentication:
      token: "{{ secret('DATABRICKS_TOKEN') }}"
    runTasks:
      - existingClusterId: <your-cluster>
        taskKey: pysparkTask
        sparkPythonTask:
          pythonFile: /Shared/hello.py
          sparkPythonTaskSource: WORKSPACE
    waitForCompletion: PT5M

Properties

runTasks*array

Min items1

The run tasks, if multiple tasks are defined you must set dependsOn on each task.

Definitions

io.kestra.plugin.databricks.job.SubmitRun-RunSubmitTaskSetting

dependsOnarray

SubTypestring

Task dependencies, set this if multiple tasks are defined on the run.

existingClusterIdstring

librariesarray

Task libraries.

io.kestra.plugin.databricks.job.task.LibrarySetting

cran

io.kestra.plugin.databricks.job.task.LibrarySetting-CranSetting

_packagestring

repostring

eggstring

jarstring

maven

io.kestra.plugin.databricks.job.task.LibrarySetting-MavenSetting

coordinatesstring

exclusionsarray

SubTypestring

repostring

pypi

io.kestra.plugin.databricks.job.task.LibrarySetting-PypiSetting

_packagestring

repostring

whlstring

notebookTask

Notebook task settings.

io.kestra.plugin.databricks.job.task.NotebookTaskSetting

baseParametersstringobject

SubTypestring

Map of task base parameters.

notebookPathstring

sourcestring

Possible Values

GITWORKSPACE

pipelineTask

Pipeline task settings.

io.kestra.plugin.databricks.job.task.PipelineTaskSetting

fullRefreshbooleanstring

pipelineIdstring

pythonWheelTask

Python Wheel task settings.

io.kestra.plugin.databricks.job.task.PythonWheelTaskSetting

entryPointstring

namedParametersstringobject

SubTypestring

Map of task named parameters.

Can be a map of string/string or a variable that binds to a JSON object.

packageNamestring

parametersstringarray

runJobTask

Run job task settings.

io.kestra.plugin.databricks.job.task.RunJobTaskSetting

jobIdstring

jobParametersobject

sparkJarTask

Spark JAR task settings.

io.kestra.plugin.databricks.job.task.SparkJarTaskSetting

jarUristring

mainClassNamestring

parametersstringarray

sparkPythonTask

Spark Python task settings.

io.kestra.plugin.databricks.job.task.SparkPythonTaskSetting

pythonFile*string

sparkPythonTaskSource*string

Possible Values

GITWORKSPACE

parametersstringarray

sparkSubmitTask

Spark Submit task settings.

io.kestra.plugin.databricks.job.task.SparkSubmitTaskSetting

parametersstringarray

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

taskKeystring

timeoutSecondsinteger

accountIdstring

Databricks account identifier.

authentication

Databricks authentication configuration.

This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.

Definitions

io.kestra.plugin.databricks.AbstractTask-AuthenticationConfig

authTypestring

azureClientIdstring

azureClientSecretstring

azureTenantIdstring

clientIdstring

clientSecretstring

googleCredentialsstring

googleServiceAccountstring

passwordstring

tokenstring

usernamestring

configFilestring

Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one.

hoststring

Databricks host.

runNamestring

The name of the run.

waitForCompletionstring

Formatduration

If set, the task will wait for the run completion.

Outputs

runIdinteger

The run identifier.

runURIstring

Formaturi

The run URI on the Databricks console.