Databricks SubmitRun

Databricks SubmitRun

Certified

Submit a Databricks run

Submits one or more tasks as an ad-hoc run; optionally waits up to waitForCompletion for terminal state. The submission is idempotent: if the Kestra worker running this task is lost and the task is resubmitted, the already in-flight or already completed Databricks run is adopted instead of a duplicate run being launched. A plain task retry after a failure still creates a genuinely new run.

yaml
type: io.kestra.plugin.databricks.job.SubmitRun

Submit a Databricks run and wait up to 5 minutes for its completion. A worker-loss resubmit adopts the same Databricks run instead of launching a duplicate one.

yaml
id: databricks_job_submit_run
namespace: company.team

tasks:
  - id: submit_run
    type: io.kestra.plugin.databricks.job.SubmitRun
    host: "{{ secret('DATABRICKS_HOST') }}"
    authentication:
      token: "{{ secret('DATABRICKS_TOKEN') }}"
    runTasks:
      - existingClusterId: <your-cluster>
        taskKey: pysparkTask
        sparkPythonTask:
          pythonFile: /Shared/hello.py
          sparkPythonTaskSource: WORKSPACE
    waitForCompletion: PT5M
Properties
Min items1

Run tasks

Task definitions for this run; set dependsOn when multiple tasks are present

Definitions
dependsOnarray
SubTypestring

Task dependencies

List of upstream taskKeys when multiple tasks run in the same submission

existingClusterIdstring
librariesarray

Task libraries

cran
_packagestring
repostring
eggstring
jarstring
maven
coordinatesstring
exclusionsarray
SubTypestring
repostring
pypi
_packagestring
repostring
whlstring
notebookTask

Notebook task settings

baseParametersstringobject
SubTypestring

Map of task base parameters.

Can be a map of string/string or a variable that binds to a JSON object.

notebookPathstring
sourcestring
Possible Values
GITWORKSPACE
pipelineTask

Pipeline task settings

fullRefreshbooleanstring
pipelineIdstring
pythonWheelTask

Python Wheel task settings

entryPointstring
namedParametersstringobject
SubTypestring

Map of task named parameters.

Can be a map of string/string or a variable that binds to a JSON object.

packageNamestring
parametersstringarray

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

runJobTask

Run job task settings

jobId*Requiredstring

Job identifier

Numeric identifier of the existing Databricks job to run

jobParametersobject
sparkJarTask

Spark JAR task settings

jarUristring
mainClassNamestring
parametersstringarray

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

sparkPythonTask

Spark Python task settings

pythonFile*Requiredstring
sparkPythonTaskSource*Requiredstring
Possible Values
GITWORKSPACE
parametersstringarray

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

sparkSubmitTask

Spark Submit task settings

parametersstringarray

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

taskKeystring
timeoutSecondsinteger

Task timeout (seconds)

Databricks account identifier

Databricks authentication configuration

This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.

Definitions
authTypestring

Authentication type

azureClientIdstring

Azure client ID

azureClientSecretstring

Azure client secret

azureTenantIdstring

Azure tenant ID

clientIdstring

Client ID

clientSecretstring

Client secret

googleCredentialsstring

Google credentials JSON

googleServiceAccountstring

Google service account email

passwordstring

Password

tokenstring

Databricks personal access token

usernamestring

Username

Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one

Databricks host

Idempotency token seed

Seed used to derive the Databricks idempotency token attached to the run submission, so that a worker-loss resubmit adopts the already in-flight or already completed run instead of launching a duplicate one. Defaults to this task run's Kestra identifier, which is unique per task execution attempt: a plain Kestra retry after a failed run still creates a new Databricks run, while a resubmit of the same attempt after a worker crash adopts the original run. Set this only if you need to key deduplication on something other than the task run itself. Two different executions sharing the same override value will cause the second one to adopt the first one's run.

Run name

Wait for completion

If set, waits up to the given duration (e.g., PT30M) for the run to finish

Formatduration

Duration

The total run duration; only set when the run has terminated

Formatdate-time

End time

When the run finished executing; only set when the run has terminated

Life cycle state

Set once the run has been submitted; only reaches a terminal value (e.g. TERMINATED, SKIPPED) when waitForCompletion is used

Result state

The run's terminal result state (e.g. SUCCESS, FAILED, TIMEDOUT); only set when the run has terminated

Run identifier

Formaturi

Run console URI

Formatdate-time

Start time

When the run started executing; only set when the run has started

State message

A human-readable description of the run's current state, useful to diagnose a non-SUCCESS result state

The duration of the Databricks run, only available when waitForCompletion is set

More Plugins in this Category

Debezium DB2
Debezium DB2

Tasks that ingest IBM Db2 change data through Debezium connectors in Kestra.

Data

3 tasks

Airbyte
Airbyte

Tasks that trigger and monitor Airbyte sync and reset jobs in self-hosted Airbyte and Airbyte Cloud

Data

4 tasks

6 blueprints