DBFS Trigger

DBFS Trigger

Certified

Trigger on new DBFS files (Legacy)

Periodically lists a DBFS path and starts one execution for files detected since the previous poll. Directories are ignored as trigger events. The trigger persists file state in Kestra's namespace KV store to avoid duplicate events. The first poll reports existing matching files as CREATE events; the default on mode is CREATE_OR_UPDATE. Recursive traversal uses explicit visited-directory tracking, while listings are materialized during each poll; keep recursively watched trees to a few thousand files to avoid excessive memory usage. Note: Databricks considers DBFS legacy; for new flows, store files in Unity Catalog Volumes instead.

yaml
type: io.kestra.plugin.databricks.dbfs.Trigger

Trigger a flow when a new file is added to a DBFS directory.

yaml
id: databricks_dbfs_trigger_flow
namespace: company.team

tasks:
  - id: each_file
    type: io.kestra.plugin.core.flow.ForEach
    values: "{{ trigger.files | jq('.[].path') }}"
    tasks:
      - id: download
        type: io.kestra.plugin.databricks.dbfs.Download
        host: "{{ secret('DATABRICKS_HOST') }}"
        authentication:
          token: "{{ secret('DATABRICKS_TOKEN') }}"
        from: "{{ taskrun.value }}"

triggers:
  - id: watch_dbfs
    type: io.kestra.plugin.databricks.dbfs.Trigger
    host: "{{ secret('DATABRICKS_HOST') }}"
    authentication:
      token: "{{ secret('DATABRICKS_TOKEN') }}"
    from: /mnt/incoming
    interval: PT1M
    recursive: true
    on: CREATE
Properties

DBFS path to watch

Absolute DBFS directory path to watch, such as /mnt/incoming. The rendered value must be an absolute DBFS path.

Databricks account identifier

DefaultNONE
Possible Values
MOVEDELETENONE

Post-detection action

NONE leaves detected files in place. MOVE relocates each detected file below moveDirectory, preserving its relative path under from. DELETE removes detected files after state is persisted. Failed actions are logged and do not cancel the trigger execution.

Defaultfalse

Specifies whether a trigger is allowed to start a new execution even if a previous run is still in progress.

Databricks authentication configuration

Configure the Databricks authentication. Any property not explicitly configured can be supplied through the standard Databricks environment variables or configuration file.

Definitions
authTypestring

Authentication type

azureClientIdstring

Azure client ID

azureClientSecretstring

Azure client secret

azureTenantIdstring

Azure tenant ID

clientIdstring

Client ID

clientSecretstring

Client secret

googleCredentialsstring

Google credentials JSON

googleServiceAccountstring

Google service account email

passwordstring

Password

tokenstring

Databricks personal access token

usernamestring

Username

Databricks configuration file

Databricks host

DefaultPT1M
Formatduration

Interval between checks

Default25

Maximum files per execution

Maximum number of detected files emitted by a single poll. Must be between 1 and 1000. The default is 25. Remaining files are evaluated on the next poll.

Move destination

Target DBFS directory when action is MOVE. For recursive polling, the relative path below from is preserved under this directory. The destination must not be inside the watched from path when recursive polling is enabled.

DefaultCREATE_OR_UPDATE
Possible Values
CREATEUPDATECREATE_OR_UPDATE

Trigger condition

Which file changes fire the trigger. Defaults to CREATE_OR_UPDATE.

Defaultfalse

Include files in subdirectories

When enabled, files below nested DBFS directories are monitored recursively. Because the listing is materialized on each poll, keep recursively watched trees to a few thousand files. When action: MOVE is used with recursive polling, moveDirectory must be outside from.

Regex pattern to match DBFS paths

Optional regular expression matched against the complete DBFS path of each file. Invalid expressions fail validation at runtime with an error naming this property.

State key

Key used to persist the trigger state. Defaults to a stable per-trigger key.

State TTL

How long the persisted trigger state is retained. Unset means no expiry.

SubTypestring
Possible Values
CREATEDSUBMITTEDRUNNINGPAUSEDRESTARTEDKILLINGSUCCESSWARNINGFAILEDKILLEDCANCELLEDQUEUEDRETRYINGRETRIEDSKIPPEDBREAKPOINTRESUBMITTED

List of execution states after which a trigger should be stopped (a.k.a. disabled).

Defaulttrue

A condition that determines whether the trigger should run.

A Pebble expression evaluated at trigger time. The trigger fires only when the expression evaluates to a truthy value (true, a non-empty string, a non-zero number). Use this to gate trigger execution on dynamic runtime values such as execution labels, flow variables, or environment conditions.

DBFS files detected since the previous poll, with their change type

Definitions
changeTypestring
Possible Values
CREATEUPDATE

Detected change type: CREATE for a newly observed file or UPDATE for changed metadata

fileSizeinteger

File size in bytes

isDirboolean

Whether the entry is a directory

modificationTimeinteger

Last modification time, in epoch milliseconds

pathstring

DBFS file path

Number of detected DBFS files

More Plugins in this Category

Metaplane
Metaplane

Run and read data-observability monitors from Metaplane.

Data

5 tasks

6 blueprints

CloudQuery
CloudQuery

Tasks that run CloudQuery commands to sync data through the CLI.

Data

2 tasks

4 blueprints