CreateJob

CreateJob
Certified
Create and run a Databricks job

yaml
type: io.kestra.plugin.databricks.job.CreateJob

Examples

yaml
id: databricks_job_create
namespace: company.team

tasks:
  - id: create_job
    type: io.kestra.plugin.databricks.job.CreateJob
    authentication:
      token: "{{ secret('DATABRICKS_TOKEN') }}"
    host: "{{ secret('DATABRICKS_HOST') }}"
    jobTasks:
      - existingClusterId: <your-cluster>
        taskKey: taskKey
        sparkPythonTask:
          pythonFile: /Shared/hello.py
          sparkPythonTaskSource: WORKSPACE
    waitForCompletion: PT5M

Properties

jobTasks *array

Min items1

Definitions

io.kestra.plugin.databricks.job.CreateJob-JobTaskSetting

dbtTask

io.kestra.plugin.databricks.job.task.DbtTaskSetting

catalogstring

commandsarray

SubTypestring

schemastring

warehouseIdstring

dependsOnarray

SubTypestring

descriptionstring

existingClusterIdstring

librariesarray

io.kestra.plugin.databricks.job.task.LibrarySetting

cran

io.kestra.plugin.databricks.job.task.LibrarySetting-CranSetting

_packagestring

repostring

eggstring

jarstring

maven

io.kestra.plugin.databricks.job.task.LibrarySetting-MavenSetting

coordinatesstring

exclusionsarray

SubTypestring

repostring

pypi

io.kestra.plugin.databricks.job.task.LibrarySetting-PypiSetting

_packagestring

repostring

whlstring

notebookTask

io.kestra.plugin.databricks.job.task.NotebookTaskSetting

baseParametersstringobject

SubTypestring

notebookPathstring

sourcestring

Possible Values

GITWORKSPACE

pipelineTask

io.kestra.plugin.databricks.job.task.PipelineTaskSetting

fullRefreshbooleanstring

pipelineIdstring

pythonWheelTask

io.kestra.plugin.databricks.job.task.PythonWheelTaskSetting

entryPointstring

namedParametersstringobject

SubTypestring

packageNamestring

parametersstringarray

runJobTask

io.kestra.plugin.databricks.job.task.RunJobTaskSetting

jobIdstring

jobParametersobject

sparkJarTask

io.kestra.plugin.databricks.job.task.SparkJarTaskSetting

jarUristring

mainClassNamestring

parametersstringarray

sparkPythonTask

io.kestra.plugin.databricks.job.task.SparkPythonTaskSetting

pythonFile*string

sparkPythonTaskSource*string

Possible Values

GITWORKSPACE

parametersstringarray

sparkSubmitTask

io.kestra.plugin.databricks.job.task.SparkSubmitTaskSetting

parametersstringarray

sqlTask

io.kestra.plugin.databricks.job.task.SqlTaskSetting

parametersstringobject

SubTypestring

queryIdstring

warehouseIdstring

taskKeystring

timeoutSecondsintegerstring

accountId string

authentication

Definitions

io.kestra.plugin.databricks.AbstractTask-AuthenticationConfig

authTypestring

azureClientIdstring

azureClientSecretstring

azureTenantIdstring

clientIdstring

clientSecretstring

googleCredentialsstring

googleServiceAccountstring

passwordstring

tokenstring

usernamestring

configFile string

host string

jobName string

pluginDefaultsRef string

waitForCompletion string

Outputs

jobId integer

jobURI string

Formaturi

runId integer

runURI string

Formaturi

DocumentDB

Tasks that manage DocumentDB collections with MongoDB-compatible insert, query, update, and delete operations.

Data

Modal

Tasks that run Modal workflows via the Modal CLI.

DataInfrastructure

CreateJobCertifiedCreate and run a Databricks job

More Plugins in this Category

DocumentDB

Modal

CreateJob
Certified
Create and run a Databricks job