PluginsAzureTriggersStorageBlobTrigger

Trigger​Trigger

Trigger Trigger

yaml
type: "io.kestra.plugin.azure.storage.blob.Trigger"

Wait for files on the Azure Blob Storage.

This trigger will poll every interval on the Azure Blob Storage. You can search for all files in a container or directory in from or you can filter the files with a regExp.The detection is atomic, internally we do a list and interact only with files listed. Once a file is detected, we download the file on internal storage and processed with declared action in order to move or delete the files from the container (to avoid double detection on new poll)

Examples

Wait for a list of files on Azure Blob Storage bucket, and then iterate through the files.

yaml
id: storage-listen
namespace: io.kestra.tests

tasks:
  - id: each
    type: io.kestra.core.tasks.flows.EachSequential
    tasks:
      - id: return
        type: io.kestra.core.tasks.debugs.Return
        format: "{{taskrun.value}}"
    value: "{{ trigger.blobs | jq('.[].uri') }}"

triggers:
  - id: watch
    type: io.kestra.plugin.azure.storage.blob.Trigger
    interval: PT5M
    endpoint: "https://yourblob.blob.core.windows.net"
    connectionString: "DefaultEndpointsProtocol=...=="
    container: "mydata"
    prefix: "trigger/storage-listen"
    action: MOVE
    moveTo: 
      container: mydata
      name: archive

Wait for a list of file on a Azure Blob Storage bucket and iterate through the files. Delete files manually after processing to prevent infinite triggering.

yaml
id: storage-listen
namespace: io.kestra.tests

tasks:
  - id: each
    type: io.kestra.core.tasks.flows.EachSequential
    tasks:
      - id: return
        type: io.kestra.core.tasks.debugs.Return
        format: "{{taskrun.value}}"
      - id: delete
        type: io.kestra.plugin.azure.storage.blob.Delete
        endpoint: "https://yourblob.blob.core.windows.net"
        connectionString: "DefaultEndpointsProtocol=...=="
        container: "mydata"
        name: "{{ taskrun.value }}"
    value: "{{ trigger.blobs | jq('.[].name') }}"

triggers:
  - id: watch
    type: io.kestra.plugin.azure.storage.blob.Trigger
    endpoint: "https://yourblob.blob.core.windows.net"
    connectionString: "DefaultEndpointsProtocol=...=="
    container: "mydata"
    prefix: "trigger/storage-listen"
    action: MOVE
    moveTo: 
      container: mydata
      name: archive

Properties

action

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
  • Possible Values:
    • MOVE
    • DELETE
    • NONE

The action to perform on the retrieved files. If using NONE, make sure to handle the files inside your flow to avoid infinite triggering.

container

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The blob container.

endpoint

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The blob service endpoint.

filter

  • Type: string
  • Dynamic: ❓
  • Required: ✔️
  • Default: FILES
  • Possible Values:
    • FILES
    • DIRECTORY
    • BOTH

The filter for files or directories.

conditions

  • Type: array
  • SubType: Condition
  • Dynamic: ❌
  • Required: ❌

List of conditions in order to limit the flow trigger.

connectionString

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Connection string of the Storage Account.

delimiter

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The delimiter for blob hierarchy, "/" for hierarchy based on directories.

interval

  • Type: string
  • Dynamic: ❌
  • Required: ❌
  • Default: 60.000000000
  • Format: duration

Interval between polling.

The interval between 2 different polls of schedule, this can avoid to overload the remote system with too many calls. For most of the triggers that depend on external systems, a minimal interval must be at least PT30S. See ISO_8601 Durations for more information of available interval values.

moveTo

The destination container and key.

prefix

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Limits the response to keys that begin with the specified prefix.

regexp

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

A regular expression to filter on the full key.

ex: regExp: .* to match all files regExp: .*2020-01-0.\\.csv to match files between 01 and 09 of january ending with .csv

sasToken

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The SAS token to use for authenticating requests.

This string should only be the query parameters (with or without a leading '?') and not a full URL.

sharedKeyAccountAccessKey

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Shared Key access key for authenticating requests.

sharedKeyAccountName

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

Shared Key account name for authenticating requests.

stopAfter

  • Type: array
  • SubType: string
  • Dynamic: ❌
  • Required: ❌

List of execution states after which a trigger should be stopped (a.k.a. disabled).

Outputs

blobs

  • Type: array
  • SubType: Blob
  • Dynamic: ❓
  • Required: ❌

The list of blobs.

Definitions

io.kestra.plugin.azure.storage.blob.models.Blob

Properties

container
  • Type: string
  • Dynamic: ❓
  • Required: ❓
name
  • Type: string
  • Dynamic: ❓
  • Required: ❓
size
  • Type: integer
  • Dynamic: ❓
  • Required: ❓
uri
  • Type: string
  • Dynamic: ❓
  • Required: ❓
  • Format: uri

io.kestra.plugin.azure.storage.blob.Copy-CopyObject

Properties

container
  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The blob container.

name
  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The full blob path on the container.