Databricks SubmitRun

Databricks SubmitRun

Certified

Submit a Databricks run

Submits one or more tasks as an ad-hoc run; optionally waits up to waitForCompletion for terminal state.

yaml
type: io.kestra.plugin.databricks.job.SubmitRun

Submit a Databricks run and wait up to 5 minutes for its completion.

yaml
id: databricks_job_submit_run
namespace: company.team

tasks:
  - id: submit_run
    type: io.kestra.plugin.databricks.job.SubmitRun
    host: "{{ secret('DATABRICKS_HOST') }}"
    authentication:
      token: "{{ secret('DATABRICKS_TOKEN') }}"
    runTasks:
      - existingClusterId: <your-cluster>
        taskKey: pysparkTask
        sparkPythonTask:
          pythonFile: /Shared/hello.py
          sparkPythonTaskSource: WORKSPACE
    waitForCompletion: PT5M
Properties
Min items1

Run tasks

Task definitions for this run; set dependsOn when multiple tasks are present

Definitions
dependsOnarray
SubTypestring

Task dependencies

List of upstream taskKeys when multiple tasks run in the same submission

existingClusterIdstring
librariesarray

Task libraries

cran
_packagestring
repostring
eggstring
jarstring
maven
coordinatesstring
exclusionsarray
SubTypestring
repostring
pypi
_packagestring
repostring
whlstring
notebookTask

Notebook task settings

baseParametersstringobject
SubTypestring

Map of task base parameters.

Can be a map of string/string or a variable that binds to a JSON object.

notebookPathstring
sourcestring
Possible Values
GITWORKSPACE
pipelineTask

Pipeline task settings

fullRefreshbooleanstring
pipelineIdstring
pythonWheelTask

Python Wheel task settings

entryPointstring
namedParametersstringobject
SubTypestring

Map of task named parameters.

Can be a map of string/string or a variable that binds to a JSON object.

packageNamestring
parametersstringarray

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

runJobTask

Run job task settings

jobIdstring
jobParametersobject
sparkJarTask

Spark JAR task settings

jarUristring
mainClassNamestring
parametersstringarray

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

sparkPythonTask

Spark Python task settings

pythonFile*Requiredstring
sparkPythonTaskSource*Requiredstring
Possible Values
GITWORKSPACE
parametersstringarray

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

sparkSubmitTask

Spark Submit task settings

parametersstringarray

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

taskKeystring
timeoutSecondsinteger

Task timeout (seconds)

Databricks account identifier

Databricks authentication configuration

This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.

Definitions
authTypestring

Authentication type

azureClientIdstring

Azure client ID

azureClientSecretstring

Azure client secret

azureTenantIdstring

Azure tenant ID

clientIdstring

Client ID

clientSecretstring

Client secret

googleCredentialsstring

Google credentials JSON

googleServiceAccountstring

Google service account email

passwordstring

Password

tokenstring

Databricks personal access token

usernamestring

Username

Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one

Databricks host

Reference (ref) of the pluginDefaults to apply to this task.

Run name

Wait for completion

If set, waits up to the given duration (e.g., PT30M) for the run to finish

Run identifier

Formaturi

Run console URI

More Plugins in this Category

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks

PostgreSQL
PostgreSQL

Tasks that connect to PostgreSQL via JDBC to run queries and loads.

Data

6 tasks

25 blueprints