AWS Download

AWS Download

Certified

Download objects from S3

Single-file mode when only key is set; multi-file mode when prefix/delimiter/regexp are provided. Saves downloads to internal storage and returns metadata per object.

yaml
type: io.kestra.plugin.aws.s3.Download
yaml
id: aws_s3_download
namespace: company.team

tasks:
  - id: download
    type: io.kestra.plugin.aws.s3.Download
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
    region: "eu-central-1"
    bucket: "my-bucket"
    key: "path/to/file"

Download a file and verify the stored S3 checksum during transfer

yaml
id: aws_s3_download_validate_checksum
namespace: company.team

tasks:
  - id: download
    type: io.kestra.plugin.aws.s3.Download
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
    region: "eu-central-1"
    bucket: "my-bucket"
    key: "path/to/file"
    validateChecksum: true
Properties

The S3 bucket name

Access Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

Defaultfalse

Compatibility mode

Use default async client (limits transfers to ~2GB) for S3-compatible endpoints.

Delimiter

Groups keys up to the delimiter; enables multi-file mode.

The endpoint with which the SDK should communicate

This property allows you to use a different S3 compatible storage backend.

The account ID of the expected bucket owner

Requests will fail with a Forbidden error (access denied) if the bucket is owned by a different account.

Force path style access

Must only be used when compatibilityMode is enabled.

Object key

Key to download in single-file mode.

Marker

Pagination start key for multi-file mode.

Default25

Max files

Limit returned files in multi-file mode; default 25.

Default1000

Max keys

Maximum keys per list request in multi-file mode; default 1000.

Reference (ref) of the pluginDefaults to apply to this task.

Prefix filter

Enables multi-file mode; downloads keys starting with this prefix.

Regexp filter

Regex on keys; enables multi-file mode.

AWS region with which the SDK should communicate

Sets the value of the RequestPayer property for this object

Secret Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

The AWS STS endpoint with which the SDKClient should communicate

AWS STS Role

The Amazon Resource Name (ARN) of the role to assume. If set the task will use the StsAssumeRoleCredentialsProvider. If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

AWS STS External Id

A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.

DefaultPT15M

AWS STS Session duration

The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.

AWS STS Session name

This property is only used when an stsRoleArn is defined.

Defaultfalse

Validate checksum after download

When true, requests S3 to return the stored checksum and the AWS SDK verifies the downloaded bytes during transfer. The object must have been uploaded with a checksum algorithm (SHA1, SHA256, CRC32, or CRC32C) for verification to occur; if the object has no stored checksum, a warning is logged and the download is not verified.

Version ID

Specific version to fetch in single-file mode.

Checksum algorithm reported by S3

One of SHA1, SHA256, CRC32, CRC32C. Null when validateChecksum was not enabled or the object has no stored checksum.

Checksum value reported by S3 (base64-encoded)

Populated when validateChecksum is true and the object has a stored checksum.

Content length (bytes)

Content type

Files

Per-key file info when multi-file mode is used.

Definitions
checksumAlgorithmstring

Checksum algorithm reported by S3

One of SHA1, SHA256, CRC32, CRC32C. Null when validateChecksum was not enabled or the object has no stored checksum.

checksumValuestring

Checksum value reported by S3 (base64-encoded)

Populated when validateChecksum is true and the object has a stored checksum.

contentLengthinteger

The size of the file in bytes

contentTypestring

The MIME type of the file

eTagstring

An ETag is an opaque identifier assigned by a web server to a specific version of a resource found at a URL

metadataobject

The metadata of the file

uristring
Formaturi

The URI of the downloaded file in Kestra's storage

versionIdstring

The version ID of the file

SubTypestring

Metadata

Formaturi

Uri

The version of the object

Unitbytes

The size of the downloaded file.

More Plugins in this Category

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks

OPA (Open Policy Agent)
OPA (Open Policy Agent)
Enterprise

Tasks to manage Open Policy Agent policies and decisions, including upload, evaluation, and partial evaluation.

Infrastructure

5 tasks

3 blueprints