Download a zip file, unzip it and upload all files in parallel to AWS S3

Loading many files into Amazon S3 one after another is slow and brittle: a large archive can hold dozens of objects, and a sequential upload leaves throughput on the table while each request waits its turn. This blueprint downloads a remote zip archive, decompresses it, and uploads every extracted file to an S3 bucket concurrently, turning a multi-step manual chore into a single declarative workflow you can schedule, retry, and reuse.

How it works

1. get_zip_file (io.kestra.plugin.core.http.Download) fetches a remote zip archive over HTTP and stores it in Kestra internal storage.
2. unzip (io.kestra.plugin.compress.ArchiveDecompress) decompresses the archive using the ZIP algorithm and exposes each extracted file through outputs.unzip.files.
3. parallel_upload_to_s3 (io.kestra.plugin.core.flow.Parallel) runs three io.kestra.plugin.aws.s3.Upload tasks at the same time, sending the CSV, PDF, and TXT files to keyed paths under powerplant/ in the target bucket.
4. A pluginDefaults block applies the AWS credentials, region, and bucket to every Upload task so you never repeat connection settings.

What you get

• Concurrent uploads instead of a slow sequential loop, cutting total runtime.
• A reusable bucket input (defaulting to declarative-data-orchestration) so the same flow targets any bucket.
• Centralized credentials via pluginDefaults, keeping each task clean.
• Full execution logs and per-task outputs for every file pushed to S3.

Who it's for

• Data engineers landing raw files into an S3 data lake.
• Platform teams building ingestion pipelines that unpack vendor archives.
• Anyone replacing ad hoc shell scripts with a versioned, auditable workflow.

Why orchestrate this with Kestra

S3 has no native scheduler or workflow engine. Kestra adds event and schedule triggers, automatic retries on transient network or API failures, parallel execution out of the box, and full lineage across the download, unzip, and upload steps. Everything is declared in readable YAML and versioned in Git, so the pipeline is reproducible rather than buried in a one-off script.

Prerequisites

• An AWS account with an existing S3 bucket in your chosen region (the flow does not create the bucket).
• Network access to the source archive URL.

Secrets

• AWS_ACCESS_KEY_ID: AWS access key with s3:PutObject permission on the target bucket.
• AWS_SECRET_ACCESS_KEY: matching AWS secret access key.
• AWS_DEFAULT_REGION: the region where the bucket lives, for example us-east-1.

Quick start

1. Add the three AWS secrets to your Kestra instance.
2. Confirm the target S3 bucket exists in AWS_DEFAULT_REGION.
3. Set the bucket input to your bucket name (or keep the default).
4. Run the flow and watch the three uploads complete in parallel.

How to extend

• Point get_zip_file at your own archive and update the file keys in the parallel tasks.
• Add more Upload tasks inside parallel_upload_to_s3 as your archive grows.
• Swap Parallel for io.kestra.plugin.core.flow.ForEach to upload a dynamic list of extracted files.
• Add a schedule or event trigger to ingest new archives automatically.

Links

• AWS plugin documentation
• Orchestrate AWS with Kestra