AWS Trigger

AWS Trigger

Certified

Trigger on S3 object creation/update

Polls a bucket on a fixed interval, listing with prefix/regex filters. Downloads matched objects to internal storage and can move or delete them to prevent reprocessing. Maintains state per trigger to avoid duplicates.

yaml
type: io.kestra.plugin.aws.s3.Trigger

Wait for a list of files on a s3 bucket and iterate through the files

yaml
id: s3_listen
namespace: company.team

tasks:
  - id: each
    type: io.kestra.plugin.core.flow.ForEach
    values: "{{ trigger.objects | jq('.[].uri') }}"
    tasks:
      - id: return
        type: io.kestra.plugin.core.debug.Return
        format: "{{ taskrun.value }}"

triggers:
  - id: watch
    type: io.kestra.plugin.aws.s3.Trigger
    interval: "PT5M"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
    region: "eu-central-1"
    bucket: "my-bucket"
    prefix: "sub-dir"
    action: MOVE
    moveTo:
      key: archive
      bucket: "new-bucket"

Wait for a list of files on a s3 bucket and iterate through the files. Delete files manually after processing to prevent infinite triggering

yaml
id: s3_listen
namespace: company.team

tasks:
  - id: each
    type: io.kestra.plugin.core.flow.ForEach
    values: "{{ trigger.objects | jq('.[].key') }}"
    tasks:
      - id: return
        type: io.kestra.plugin.core.debug.Return
        format: "{{ taskrun.value }}"

      - id: delete
        type: io.kestra.plugin.aws.s3.Delete
        accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
        secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
        region: "eu-central-1"
        bucket: "my-bucket"
        key: "{{ taskrun.value }}"

triggers:
  - id: watch
    type: io.kestra.plugin.aws.s3.Trigger
    interval: "PT5M"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
    region: "eu-central-1"
    bucket: "my-bucket"
    prefix: "sub-dir"
    action: NONE
Properties
Possible Values
MOVEDELETENONE

The action to perform on the retrieved files. If using 'NONE' make sure to handle the files inside your flow to avoid infinite triggering

The S3 bucket where to download the file

Access Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

Defaultfalse

Specifies whether a trigger is allowed to start a new execution even if a previous run is still in progress.

Defaultfalse

Enable compatibility mode

Use it to connect to S3 bucket with S3 compatible services that don't support the new transport client.

A delimiter is a character you use to group keys

The EncodingType property for this object

The endpoint with which the SDK should communicate

This property allows you to use a different S3 compatible storage backend.

The account ID of the expected bucket owner

If the bucket is owned by a different account, the request fails with the HTTP status code 403 Forbidden (access denied).

DefaultBOTH
Possible Values
FILESDIRECTORYBOTH

The type of objects to filter: files, directory, or both

Defaultfalse

Force path style access

Must only be used when compatibilityMode is enabled.

DefaultPT1M
Formatduration

Interval between polling.

The interval between 2 different polls of schedule, this can avoid to overload the remote system with too many calls. For most of the triggers that depend on external systems, a minimal interval must be at least PT30S. See ISO_8601 Durations for more information of available interval values.

Marker is where you want Amazon S3 to start listing from

Amazon S3 starts listing after this specified key. Marker can be any key in the bucket.

Default25

The maximum number of files to retrieve at once

Default1000

Maximum number of objects returned across all paginated S3 calls

Total upper bound on the objects returned. It also sets the per-page size sent to S3, which AWS caps at 1000. The listing pages until this total is reached or the bucket is exhausted. Default 1000.

The destination bucket and key for MOVE action

Definitions
bucket*Requiredstring

Bucket

key*Requiredstring

Key

kmsKeyIdstring

KMS Key ARN or Key ID to use when server side encryption is AWS_KMS

serverSideEncryptionstring
Possible Values
NONEAES256AWS_KMS

Server side encryption to apply to the target object

Example: AES256 or AWS_KMS

DefaultCREATE_OR_UPDATE
Possible Values
CREATEUPDATECREATE_OR_UPDATE

Trigger condition

Which object events fire the trigger; defaults to CREATE_OR_UPDATE.

Limits the response to keys that begin with the specified prefix

A regexp to filter on full key

ex: regExp: .* to match all files regExp: .*2020-01-0.\\.csv to match files between 01 and 09 of january ending with .csv

AWS region with which the SDK should communicate

Sets the value of the RequestPayer property for this object

Secret Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

State key

Key under which the trigger persists its dedup state; defaults to a stable per-trigger value.

State TTL

How long persisted dedup state is retained before matched objects can trigger again; unset means no expiry.

SubTypestring
Possible Values
CREATEDSUBMITTEDRUNNINGPAUSEDRESTARTEDKILLINGSUCCESSWARNINGFAILEDKILLEDCANCELLEDQUEUEDRETRYINGRETRIEDSKIPPEDBREAKPOINTRESUBMITTED

List of execution states after which a trigger should be stopped (a.k.a. disabled).

The AWS STS endpoint with which the SDKClient should communicate

AWS STS Role

The Amazon Resource Name (ARN) of the role to assume. If set the task will use the StsAssumeRoleCredentialsProvider. If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

AWS STS External Id

A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.

DefaultPT15M

AWS STS Session duration

The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.

AWS STS Session name

This property is only used when an stsRoleArn is defined.

Defaulttrue

A condition that determines whether the trigger should run.

A Pebble expression evaluated at trigger time. The trigger fires only when the expression evaluates to a truthy value (true, a non-empty string, a non-zero number). Use this to gate trigger execution on dynamic runtime values such as execution labels, flow variables, or environment conditions.

List of S3 objects that triggered the flow, each with its change type

Definitions
changeTypestring
Possible Values
CREATEUPDATE
checksumAlgorithmstring

Checksum algorithm

The checksum algorithm used for the object, if any.

checksumValuestring

Checksum value

The object's checksum value, if any.

etagstring

ETag

The entity tag (ETag) of the object.

keystring

Key

The object key within the bucket.

lastModifiedstring
Formatdate-time

Last modified

Timestamp of the object's last modification.

owner

Owner

The object owner.

displayNamestring

Display name

The S3 owner's display name.

idstring

Owner ID

The S3 owner's canonical user ID.

sizeinteger

Size

The object size in bytes.

uristring
Formaturi

URI

Internal storage URI of the downloaded object, when applicable.

More Plugins in this Category

Azure (EE)
Azure
Enterprise

Tasks that integrate Microsoft Azure services into Kestra enterprise workflows.

CloudDataInfrastructure

7 tasks

Prefect
Prefect

Tasks that trigger Prefect deployments and monitor flow runs.

Data

1 tasks

1 blueprints