Databricks CreateJob

Databricks CreateJob

Certified

Create and run a Databricks job

Creates a Databricks job with one or more tasks, submits it immediately, and optionally waits for completion. Use numWorkers/cluster settings inside each task; set waitForCompletion (ISO-8601 duration) to block until the run ends.

yaml
type: io.kestra.plugin.databricks.job.CreateJob

Create a Databricks job, run it, and wait for completion for five minutes.

yaml
id: databricks_job_create
namespace: company.team

tasks:
  - id: create_job
    type: io.kestra.plugin.databricks.job.CreateJob
    authentication:
      token: "{{ secret('DATABRICKS_TOKEN') }}"
    host: "{{ secret('DATABRICKS_HOST') }}"
    jobTasks:
      - existingClusterId: <your-cluster>
        taskKey: taskKey
        sparkPythonTask:
          pythonFile: /Shared/hello.py
          sparkPythonTaskSource: WORKSPACE
    waitForCompletion: PT5M
Properties
Min items1

Job tasks

Task definitions; when multiple tasks are present, specify dependsOn for ordering

Definitions
dbtTask

DBT task settings

catalogstring
commandsarray
SubTypestring
schemastring
warehouseIdstring
dependsOnarray
SubTypestring

Task dependencies

List of upstream taskKeys when multiple tasks run in the job

descriptionstring

Task description

existingClusterIdstring

Existing cluster ID

Cluster to reuse for this task; omit to use task-specific settings

librariesarray

Task libraries

cran
_packagestring
repostring
eggstring
jarstring
maven
coordinatesstring
exclusionsarray
SubTypestring
repostring
pypi
_packagestring
repostring
whlstring
notebookTask

Notebook task settings

baseParametersstringobject
SubTypestring

Map of task base parameters.

Can be a map of string/string or a variable that binds to a JSON object.

notebookPathstring
sourcestring
Possible Values
GITWORKSPACE
pipelineTask

Pipeline task settings

fullRefreshbooleanstring
pipelineIdstring
pythonWheelTask

Python Wheel task settings

entryPointstring
namedParametersstringobject
SubTypestring

Map of task named parameters.

Can be a map of string/string or a variable that binds to a JSON object.

packageNamestring
parametersstringarray

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

runJobTask

Run job task settings

jobIdstring
jobParametersobject
sparkJarTask

Spark JAR task settings

jarUristring
mainClassNamestring
parametersstringarray

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

sparkPythonTask

Spark Python task settings

pythonFile*Requiredstring
sparkPythonTaskSource*Requiredstring
Possible Values
GITWORKSPACE
parametersstringarray

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

sparkSubmitTask

Spark Submit task settings

parametersstringarray

List of task parameters.

Can be a list of strings or a variable that binds to a JSON array of strings.

sqlTask

SQL task settings

parametersstringobject
SubTypestring

Map of task parameters.

Can be a map of string/string or a variable that binds to a JSON object.

queryIdstring
warehouseIdstring
taskKeystring

Task key

Unique key per task; required when multiple tasks are defined

timeoutSecondsintegerstring

Task timeout (seconds)

Databricks account identifier

Databricks authentication configuration

This property allows to configure the authentication to Databricks, different properties should be set depending on the type of authentication and the cloud provider. All configuration options can also be set using the standard Databricks environment variables. Check the Databricks authentication guide for more information.

Definitions
authTypestring

Authentication type

azureClientIdstring

Azure client ID

azureClientSecretstring

Azure client secret

azureTenantIdstring

Azure tenant ID

clientIdstring

Client ID

clientSecretstring

Client secret

googleCredentialsstring

Google credentials JSON

googleServiceAccountstring

Google service account email

passwordstring

Password

tokenstring

Databricks personal access token

usernamestring

Username

Databricks configuration file, use this if you don't want to configure each Databricks account properties one by one

Databricks host

Job name

Reference (ref) of the pluginDefaults to apply to this task.

Wait for completion

If set, waits up to the given duration (e.g., PT1H) for the submitted run to finish

Job identifier

Formaturi

Job console URI

Run identifier

Formaturi

Run console URI

More Plugins in this Category

Clickhouse
Clickhouse

Tasks that connect to ClickHouse via JDBC to run queries and loads.

Data

5 tasks

2 blueprints

Weaviate
Weaviate

Tasks that manage and query vectors in Weaviate.

DataAI

4 tasks

2 blueprints