AWS S3 Event Trigger

Run an event-driven data pipeline that fires the moment new files arrive in an Amazon S3 bucket. Instead of polling on a fixed cron or wiring up bespoke Lambda glue, this blueprint uses Kestra's native io.kestra.plugin.aws.s3.Trigger to watch a bucket prefix and start a flow run for each batch of detected objects, so ingestion begins as soon as data is available. It solves the classic problem of reacting to file drops in object storage without standing up extra infrastructure.

How it works

1. The wait_for_s3_object trigger (io.kestra.plugin.aws.s3.Trigger) polls the declarative-orchestration bucket under the demo prefix every second (interval: PT1S), filtering for files only (filter: FILES).
2. When new objects are detected, the trigger fires a flow run and exposes the matched objects on {{ trigger.objects }}.
3. To guarantee exactly-once processing, the trigger archives each consumed object using action: MOVE and moveTo the archive/demo/ key, so the same file is never picked up twice.
4. The each task (io.kestra.plugin.core.flow.ForEach) iterates over the object URIs extracted with jq('.[].uri'), with concurrencyLimit: 0 to fan out across all objects at once.
5. For each item, the s3_object task (io.kestra.plugin.core.debug.Return) emits the object URI, a placeholder you swap for your real downstream processing.

What you get

• Event-driven ingestion that reacts to S3 file drops in near real time.
• Exactly-once handling via automatic archive-on-consume.
• Parallel processing of every object in a batch.
• A clear template to plug in transforms, loads, or notifications.

Who it's for

• Data engineers building ingestion pipelines on AWS.
• Platform teams replacing ad hoc Lambda or cron-based file watchers.
• Analytics teams who need fresh data the moment it lands in S3.

Why orchestrate this with Kestra

S3 event notifications can push to SQS or Lambda, but they do not give you orchestration: no retries, no per-run lineage, no parallel fan-out, and no declarative pipeline you can version in Git. Kestra fills that gap with an event trigger that starts a fully observable flow, automatic retries, execution history, and a declarative YAML definition that any engineer can read and extend.

Prerequisites

• An AWS account with an S3 bucket and read plus write access to the target prefix.

Secrets

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_DEFAULT_REGION

Quick start

1. Add the three AWS secrets above to your Kestra instance.
2. Update bucket and prefix on the trigger to match your bucket and folder.
3. Adjust moveTo to the archive location you want for consumed files.
4. Add the flow, then drop a file into the watched prefix and watch the execution start.

How to extend

• Replace the s3_object Return task with a download, transform, or load step (for example, write to a warehouse or DuckDB).
• Tune interval for less aggressive polling, or change action to NONE to leave objects in place.
• Add errors tasks and retries for resilient ingestion.
• Fan results into downstream flows with subflow tasks for multi-stage pipelines.

Links

• Plugin docs: https://kestra.io/plugins/plugin-aws
• Orchestrate AWS with Kestra: https://kestra.io/orchestration/aws