CreateJob
Create a Databricks job and run it.

Create a Databricks job and run it.

Set waitForCompletion to the desired maximum duration if you want the task to wait for the job completion (e.g., PT1H to wait up to one hour).

yaml
type: "io.kestra.plugin.databricks.job.CreateJob"

Examples

Create a Databricks job, run it, and wait for completion for five minutes.

yaml
id: databricks_job_create
namespace: company.team

tasks:
  - id: create_job
    type: io.kestra.plugin.databricks.job.CreateJob
    authentication:
      token: "{{ secret('DATABRICKS_TOKEN') }}"
    host: "{{ secret('DATABRICKS_HOST') }}"
    jobTasks:
      - existingClusterId: <your-cluster>
        taskKey: taskKey
        sparkPythonTask:
          pythonFile: /Shared/hello.py
          sparkPythonTaskSource: WORKSPACE
    waitForCompletion: PT5M

Properties

jobTasks*array

Min items1

The job tasks, if multiple tasks are defined you must set dependsOn on each task.

Definitions

io.kestra.plugin.databricks.job.CreateJob-JobTaskSetting

dbtTask

DBT task settings.

io.kestra.plugin.databricks.job.task.DbtTaskSetting

catalogstring

commandsarray

SubTypestring

schemastring

warehouseIdstring

dependsOnarray

SubTypestring

Task dependencies, set this if multiple tasks are defined on the job.

descriptionstring

Task description.

existingClusterIdstring

The identifier of the cluster.

librariesarray

Task libraries.

io.kestra.plugin.databricks.job.task.LibrarySetting

cran

io.kestra.plugin.databricks.job.task.LibrarySetting-CranSetting

_packagestring

repostring

eggstring

jarstring

maven

io.kestra.plugin.databricks.job.task.LibrarySetting-MavenSetting

coordinatesstring

exclusionsarray

SubTypestring

repostring

pypi

io.kestra.plugin.databricks.job.task.LibrarySetting-PypiSetting

_packagestring

repostring

whlstring

notebookTask

Notebook task settings.

io.kestra.plugin.databricks.job.task.NotebookTaskSetting

baseParametersstringobject

SubTypestring

Map of task base parameters.

notebookPathstring

sourcestring

Possible Values

GITWORKSPACE

pipelineTask

Pipeline task settings.

io.kestra.plugin.databricks.job.task.PipelineTaskSetting

fullRefreshbooleanstring

pipelineIdstring

pythonWheelTask

Python Wheel task settings.

io.kestra.plugin.databricks.job.task.PythonWheelTaskSetting

entryPointstring

namedParametersstringobject

SubTypestring

Map of task named parameters.

packageNamestring

parametersstringarray

runJobTask

Run job task settings.

io.kestra.plugin.databricks.job.task.RunJobTaskSetting

jobIdstring

jobParametersobject

sparkJarTask

Spark JAR task settings.

io.kestra.plugin.databricks.job.task.SparkJarTaskSetting

jarUristring

mainClassNamestring

parametersstringarray

sparkPythonTask

Spark Python task settings.

io.kestra.plugin.databricks.job.task.SparkPythonTaskSetting

pythonFile*string

sparkPythonTaskSource*string

Possible Values

GITWORKSPACE

parametersstringarray

sparkSubmitTask

Spark Submit task settings.

io.kestra.plugin.databricks.job.task.SparkSubmitTaskSetting

parametersstringarray

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

sqlTask

SQL task settings.

io.kestra.plugin.databricks.job.task.SqlTaskSetting

parametersstringobject

SubTypestring

Map of task parameters.

Can be a map of string/string or a variable that binds to a JSON object.

queryIdstring

warehouseIdstring

taskKeystring

Task key.

timeoutSecondsintegerstring

Task timeout in seconds.

accountIdstring

Databricks account identifier.

authentication

Databricks authentication configuration.

This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.

Definitions

io.kestra.plugin.databricks.AbstractTask-AuthenticationConfig

authTypestring

azureClientIdstring

azureClientSecretstring

azureTenantIdstring

clientIdstring

clientSecretstring

googleCredentialsstring

googleServiceAccountstring

passwordstring

tokenstring

usernamestring

configFilestring

Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one.

hoststring

Databricks host.

jobNamestring

The name of the job.

waitForCompletionstring

Formatduration

If set, the task will wait for the job run completion for up to the waitForCompletion duration before timing out.

Outputs

jobIdinteger

The job identifier.

jobURIstring

Formaturi

The job URI on the Databricks console.

runIdinteger

The run identifier.

runURIstring

Formaturi

The run URI on the Databricks console.

Run a task on an on-demand Databricks cluster

Core Plugins and tasks

Tasks that provide Kestra's built-in orchestration, I/O, and observability capabilities.

CoreStorageScript

Azure

Enterprise

Tasks that integrate Microsoft Azure services into Kestra enterprise workflows.

AlertingCloudStorageBatch

DBT

Integrate dbt data transformations into Kestra orchestration pipelines.

Tool

CreateJob
Create a Databricks job and run it.

Create automations with Databricks JOB CreateJob

Run a task on an on-demand Databricks cluster

More Plugins in this Category

Core Plugins and tasks

Azure

DBT

1.1.0

CreateJobCreate a Databricks job and run it.

Create automations with Databricks JOB CreateJob

Run a task on an on-demand Databricks cluster

More Plugins in this Category

Core Plugins and tasks

Azure

DBT

1.1.0

CreateJob
Create a Databricks job and run it.