Azure Trigger

Azure Trigger

Certified

Trigger a flow on new file arrival in Azure Data Lake Storage

This trigger will poll the specified Azure Data Lake Storage file system every interval. Using the from and regExp properties, you can define which files' arrival will trigger the flow. Under the hood, we use the Azure Data Lake Storage API to list the files in a specified location and download them to the internal storage and process them with the declared action. You can use the action property to move or delete the files from the container after processing to avoid the trigger to be fired again for the same files during the next polling interval.

yaml
type: io.kestra.plugin.azure.storage.adls.Trigger

Run a flow if one or more files arrived in the specified Azure Data Lake Storage file system location. Then, process all files in a for-loop either sequentially or concurrently, depending on the concurrencyLimit property.

yaml
id: react_to_files
namespace: company.team

tasks:
  - id: each
    type: io.kestra.plugin.core.flow.Loop
    concurrencyLimit: 1
    values: "{{ trigger.files | jq('.[].uri') }}"
    tasks:
      - id: return
        type: io.kestra.plugin.core.debug.Return
        format: "{{ item.value }}"

triggers:
  - id: watch
    type: io.kestra.plugin.azure.storage.adls.Trigger
    interval: PT5M
    endpoint: "https://yourblob.blob.core.windows.net"
    connectionString: "{{ secret('AZURE_CONNECTION_STRING') }}"
    fileSystem: myFileSystem
    directoryPath: yourDirectory/subdirectory
Properties
DefaultNONE
Possible Values
MOVEDELETENONE

The action to perform on the retrieved files. If using NONE, make sure to handle the files inside your flow to avoid infinite triggering

Defaultfalse

Specifies whether a trigger is allowed to start a new execution even if a previous run is still in progress.

Connection string of the Storage Account.

The directory path to monitor

The blob service endpoint.

The ADLS file system (container) to monitor

DefaultPT1M
Formatduration

Interval between polling.

The interval between 2 different polls of schedule, this can avoid to overload the remote system with too many calls. For most of the triggers that depend on external systems, a minimal interval must be at least PT30S. See ISO_8601 Durations for more information of available interval values.

Default25

The maximum number of files to retrieve at once

Limits the number of files retrieved per polling interval. If not specified, all matching files will be retrieved.

The destination container and key

Definitions
directoryPath*Requiredstring

The full destination directory path on the file system

fileSystem*Requiredstring

The destination file system

DefaultCREATE_OR_UPDATE
Possible Values
CREATEUPDATECREATE_OR_UPDATE

Trigger event type

Defines when the trigger fires.

  • CREATE: only for newly discovered entities.
  • UPDATE: only when an already-seen entity changes.
  • CREATE_OR_UPDATE: fires on either event.

The SAS token to use for authenticating requests.

This string should only be the query parameters (with or without a leading '?') and not a full URL.

Shared Key access key for authenticating requests.

Shared Key account name for authenticating requests.

State key

JSON-type KV key for persisted state. Default: <namespace>__<flowId>__<triggerId>

State TTL

TTL for persisted state entries (e.g., PT24H, P7D).

SubTypestring
Possible Values
CREATEDSUBMITTEDRUNNINGPAUSEDRESTARTEDKILLINGSUCCESSWARNINGFAILEDKILLEDCANCELLEDQUEUEDRETRYINGRETRIEDSKIPPEDBREAKPOINTRESUBMITTED

List of execution states after which a trigger should be stopped (a.k.a. disabled).

Defaulttrue

A condition that determines whether the trigger should run.

A Pebble expression evaluated at trigger time. The trigger fires only when the expression evaluates to a truthy value (true, a non-empty string, a non-zero number). Use this to gate trigger execution on dynamic runtime values such as execution labels, flow variables, or environment conditions.

List of files that triggered the flow, each with its change type

Definitions
changeTypestring
Possible Values
CREATEUPDATE
accessControlListarray
SubTypestring
archiveStatus
archiveTier
contentEncodingstring
contentLanguagestring
contentMd5string
contentTypestring
creationTimestring
Formatdate-time
eTagstring
fileNamestring
fileSystemstring
groupstring
isDirectoryboolean
lastModifedstring
Formatdate-time
leaseDurationstring
Possible Values
INFINITEFIXED
leaseStatestring
Possible Values
AVAILABLELEASEDEXPIREDBREAKINGBROKEN
leaseStatusstring
Possible Values
LOCKEDUNLOCKED
namestring
ownerstring
permissionsstring
sizeinteger
uristring
Formaturi

More Plugins in this Category

Prefect
Prefect

Tasks that trigger Prefect deployments and monitor flow runs.

Data

1 tasks

1 blueprints

Azure (EE)
Azure
Enterprise

Tasks that integrate Microsoft Azure services into Kestra enterprise workflows.

CloudDataInfrastructure

7 tasks