CreateJob

CreateJob

Certified

Create and run a Databricks job

yaml
type: io.kestra.plugin.databricks.job.CreateJob
yaml
id: databricks_job_create
namespace: company.team

tasks:
  - id: create_job
    type: io.kestra.plugin.databricks.job.CreateJob
    authentication:
      token: "{{ secret('DATABRICKS_TOKEN') }}"
    host: "{{ secret('DATABRICKS_HOST') }}"
    jobTasks:
      - existingClusterId: <your-cluster>
        taskKey: taskKey
        sparkPythonTask:
          pythonFile: /Shared/hello.py
          sparkPythonTaskSource: WORKSPACE
    waitForCompletion: PT5M
Properties
Min items1
Definitions
dbtTask
catalogstring
commandsarray
SubTypestring
schemastring
warehouseIdstring
dependsOnarray
SubTypestring
descriptionstring
existingClusterIdstring
librariesarray
cran
_packagestring
repostring
eggstring
jarstring
maven
coordinatesstring
exclusionsarray
SubTypestring
repostring
pypi
_packagestring
repostring
whlstring
notebookTask
baseParametersstringobject
SubTypestring
notebookPathstring
sourcestring
Possible Values
GITWORKSPACE
pipelineTask
fullRefreshbooleanstring
pipelineIdstring
pythonWheelTask
entryPointstring
namedParametersstringobject
SubTypestring
packageNamestring
parametersstringarray
runJobTask
jobIdstring
jobParametersobject
sparkJarTask
jarUristring
mainClassNamestring
parametersstringarray
sparkPythonTask
pythonFile*Requiredstring
sparkPythonTaskSource*Requiredstring
Possible Values
GITWORKSPACE
parametersstringarray
sparkSubmitTask
parametersstringarray
sqlTask
parametersstringobject
SubTypestring
queryIdstring
warehouseIdstring
taskKeystring
timeoutSecondsintegerstring
Definitions
authTypestring
azureClientIdstring
azureClientSecretstring
azureTenantIdstring
clientIdstring
clientSecretstring
googleCredentialsstring
googleServiceAccountstring
passwordstring
tokenstring
usernamestring
Formaturi
Formaturi

More Plugins in this Category

MongoDB
MongoDB

Tasks that query and manipulate MongoDB collections.

Data

8 tasks

3 blueprints

Transform
Transform

Transform typed records, variables, and Ion data.

CoreData

6 tasks