Blueprints
Download a zip file, unzip it and upload all files in parallel to AWS S3 - using pluginDefaults to avoid boilerplate code
Source
yaml
id: s3-parallel-uploads
namespace: company.team
inputs:
- id: bucket
type: STRING
defaults: declarative-data-orchestration
tasks:
- id: get_zip_file
type: io.kestra.plugin.core.http.Download
uri: https://wri-dataportal-prod.s3.amazonaws.com/manual/global_power_plant_database_v_1_3.zip
- id: unzip
type: io.kestra.plugin.compress.ArchiveDecompress
algorithm: ZIP
from: "{{ outputs.get_zip_file.uri }}"
- id: parallel_upload_to_s3
type: io.kestra.plugin.core.flow.Parallel
tasks:
- id: csv
type: io.kestra.plugin.aws.s3.Upload
from: "{{ outputs.unzip.files['global_power_plant_database.csv'] }}"
key: powerplant/global_power_plant_database.csv
- id: pdf
type: io.kestra.plugin.aws.s3.Upload
from: "{{
outputs.unzip.files['Estimating_Power_Plant_Generation_in_the_Global_P\
ower_Plant_Database.pdf'] }}"
key: powerplant/Estimating_Power_Plant_Generation_in_the_Global_Power_Plant_Database.pdf
- id: txt
type: io.kestra.plugin.aws.s3.Upload
from: "{{ outputs.unzip.files['RELEASE_NOTES.txt'] }}"
key: powerplant/RELEASE_NOTES.txt
pluginDefaults:
- type: io.kestra.plugin.aws.s3.Upload
values:
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
region: "{{ secret('AWS_DEFAULT_REGION') }}"
bucket: "{{ inputs.bucket }}"
About this blueprint
S3 Parallel Inputs Software Engineering Outputs
This flow downloads a zip file, unzips it, and uploads all files to S3 in parallel.
This flow assumes AWS credentials stored as secrets AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION configured using pluginDefaults to avoid boilerplate configuration.
The flow does not create the S3 bucket, and assumes that the bucket name provided in the inputs already exists in the AWS_DEFAULT_REGION.