List objects in an S3 bucket and process them in parallel

List objects stored in an Amazon S3 bucket under a given prefix and fan out over every returned file in parallel. This blueprint solves a common S3 data engineering problem: you have a folder of objects (logs, exports, CSVs, partitioned data) and you need to iterate over each one to validate, transform, or route it, without writing brittle glue code. Kestra reads the live S3 listing at runtime and spawns one task run per object, so the loop always matches what is actually in the bucket.

How it works

1. The list_objects task (io.kestra.plugin.aws.s3.List) queries the bucket from the bucket input and filters to the prefix powerplant/, returning the matching object metadata.
2. The print_objects task (io.kestra.plugin.core.log.Log) logs the full list of objects found, giving you a quick audit of what was discovered.
3. The map_over_s3_objects task (io.kestra.plugin.core.flow.ForEach) iterates over outputs.list_objects.objects with concurrencyLimit: 0, running all items in parallel.
4. Inside the loop, the filename task logs each object's key and size by parsing taskrun.value with the json() function.

What you get

• A dynamic, data-driven loop that always matches the current bucket contents.
• Parallel processing of every object with a configurable concurrency limit.
• Per-object visibility (key and size) in the execution logs and outputs.
• A reusable scaffold for any per-file S3 workload.

Who it's for

• Data engineers processing batches of files landing in S3.
• Platform teams building ingestion and validation pipelines.
• Anyone replacing ad hoc S3 iteration scripts with declarative orchestration.

Why orchestrate this with Kestra

S3 stores your objects but does not orchestrate work over them. Kestra adds event triggers (for example, run on a schedule or when new objects arrive), automatic retries on transient AWS errors, full execution lineage and logs, and a declarative YAML definition you can version control. The dynamic ForEach fan-out and per-task concurrency control fill the gap that a bucket alone cannot: turning a static object listing into governed, observable, parallel processing.

Prerequisites

• An AWS account with an S3 bucket and read access to the target prefix.
• A Kestra instance with the AWS plugin available.

Secrets

• AWS_ACCESS_KEY_ID: AWS access key with S3 read permissions.
• AWS_SECRET_ACCESS_KEY: AWS secret access key.
• AWS_DEFAULT_REGION: AWS region of the bucket (for example, us-east-1).

Quick start

1. Add the three AWS secrets above to your Kestra instance.
2. Set the bucket input to your bucket name (default declarative-data-orchestration).
3. Adjust the prefix on list_objects to match your folder.
4. Execute the flow and review the logged objects and per-file output.

How to extend

• Replace the inner filename log with a real per-object task: download, transform, copy, or load into a warehouse.
• Add a Schedule or S3 event trigger to run automatically.
• Tune concurrencyLimit to cap parallelism against AWS rate limits.
• Filter objects by size or extension before the loop.

Links

• AWS plugin documentation
• Orchestrate AWS with Kestra