AWS Downloads
CertifiedDownload multiple S3 objects
AWS Downloads
Download multiple S3 objects
Lists objects with filters, downloads them in bulk, emits metrics, and optionally performs a post-action (move/delete).
type: io.kestra.plugin.aws.s3.DownloadsExamples
id: aws_s3_downloads
namespace: company.team
tasks:
- id: downloads
type: io.kestra.plugin.aws.s3.Downloads
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: "eu-central-1"
bucket: "my-bucket"
prefix: "sub-dir"
Download a prefix and verify the stored S3 checksum on each object
id: aws_s3_downloads_validate_checksum
namespace: company.team
tasks:
- id: downloads
type: io.kestra.plugin.aws.s3.Downloads
accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
region: "eu-central-1"
bucket: "my-bucket"
prefix: "sub-dir"
validateChecksum: true
Properties
action *Requiredstring
MOVEDELETENONEThe action to perform on the retrieved files. If using 'NONE' make sure to handle the files inside your flow to avoid infinite triggering
bucket *Requiredstring
The S3 bucket where to download the file
accessKeyId string
Access Key Id in order to connect to AWS
If no credentials are defined, we will use the default credentials provider chain to fetch credentials.
compatibilityMode booleanstring
falseCompatibility mode
Use default async client for S3-compatible endpoints (limits transfers to ~2GB).
delimiter string
A delimiter is a character you use to group keys
encodingType string
The EncodingType property for this object
endpointOverride string
The endpoint with which the SDK should communicate
This property allows you to use a different S3 compatible storage backend.
expectedBucketOwner string
The account ID of the expected bucket owner
If the bucket is owned by a different account, the request fails with the HTTP status code 403 Forbidden (access denied).
filter string
BOTHFILESDIRECTORYBOTHThe type of objects to filter: files, directory, or both
forcePathStyle booleanstring
Force path style access
Must only be used when compatibilityMode is enabled.
marker string
Marker is where you want Amazon S3 to start listing from
Amazon S3 starts listing after this specified key. Marker can be any key in the bucket.
maxFiles integerstring
25Max files
Limit returned files; default 25.
maxKeys integerstring
1000Maximum number of objects returned across all paginated S3 calls
Total upper bound on the objects returned. It also sets the per-page size sent to S3, which AWS caps at 1000. The listing pages until this total is reached or the bucket is exhausted. Default 1000.
moveTo
The destination bucket and key for MOVE action
io.kestra.plugin.aws.s3.Copy-CopyObject
Bucket
Key
KMS Key ARN or Key ID to use when server side encryption is AWS_KMS
NONEAES256AWS_KMSServer side encryption to apply to the target object
Example: AES256 or AWS_KMS
pluginDefaultsRef Non-dynamicstring
Reference (ref) of the pluginDefaults to apply to this task.
prefix string
Limits the response to keys that begin with the specified prefix
regexp string
A regexp to filter on full key
ex:
regExp: .* to match all files
regExp: .*2020-01-0.\\.csv to match files between 01 and 09 of january ending with .csv
region string
AWS region with which the SDK should communicate
requestPayer string
Sets the value of the RequestPayer property for this object
secretKeyId string
Secret Key Id in order to connect to AWS
If no credentials are defined, we will use the default credentials provider chain to fetch credentials.
sessionToken string
AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource
If no credentials are defined, we will use the default credentials provider chain to fetch credentials.
stsEndpointOverride string
The AWS STS endpoint with which the SDKClient should communicate
stsRoleArn string
AWS STS Role
The Amazon Resource Name (ARN) of the role to assume. If set the task will use the StsAssumeRoleCredentialsProvider. If no credentials are defined, we will use the default credentials provider chain to fetch credentials.
stsRoleExternalId string
AWS STS External Id
A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.
stsRoleSessionDuration string
PT15MAWS STS Session duration
The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.
stsRoleSessionName string
AWS STS Session name
This property is only used when an stsRoleArn is defined.
validateChecksum booleanstring
falseValidate checksum after download
When true, requests S3 to return the stored checksum and the AWS SDK verifies the downloaded bytes during transfer. Each object must have been uploaded with a checksum algorithm (SHA1, SHA256, CRC32, or CRC32C) for verification to occur; if an object has no stored checksum, a warning is logged and the download is not verified.
Outputs
objects array
Objects
Downloaded objects with metadata.
io.kestra.plugin.aws.s3.models.S3Object
date-timeio.kestra.plugin.aws.s3.models.Owner
urioutputFiles object
Output files
Map of object key to downloaded URI.
Metrics
files.count counter
objectsThe number of files downloaded from the S3 bucket.
files.size.total counter
bytesThe total size in bytes of all downloaded files.
