AWS Upload

AWS Upload
Certified
Upload files to S3

Uploads one or many files to an S3 bucket. Accepts inputs as URIs, lists, or maps. Supports metadata, ACLs, SSE, checksum, and Object Lock options. Compatibility mode enables S3-compatible endpoints but limits size to ~2GB.

yaml
type: io.kestra.plugin.aws.s3.Upload

Examples

Upload a FILE input to S3

yaml
id: aws_s3_upload
namespace: company.team

inputs:
  - id: bucket
    type: STRING
    defaults: my-bucket

  - id: myfile
    type: FILE

tasks:
  - id: upload
    type: io.kestra.plugin.aws.s3.Upload
    region: "{{ secret('AWS_DEFAULT_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
    from: "{{ inputs.myfile }}"
    bucket: "{{ inputs.bucket }}"
    key: "path/to/file"

Download a file and upload it to S3

yaml
id: upload_file_to_s3
namespace: company.team

inputs:
  - id: bucket
    type: STRING
    defaults: my-bucket

  - id: file_url
    type: STRING
    defaults: https://wri-dataportal-prod.s3.amazonaws.com/manual/global_power_plant_database_v_1_3.zip

tasks:
  - id: download_file
    type: io.kestra.plugin.core.http.Download
    uri: "{{ inputs.file_url }}"

  - id: upload_to_s3
    type: io.kestra.plugin.aws.s3.Upload
    from: "{{ outputs.download_file.uri }}"
    key: powerplant/global_power_plant_database.zip
    bucket: "{{ inputs.bucket }}"
    region: "{{ secret('AWS_DEFAULT_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"

Upload multiple files to S3 using a JSON array

yaml
id: upload_multiple_files_from_json_array
namespace: company.team

inputs:
  - id: bucket
    type: STRING
    defaults: my-bucket

tasks:
  - id: download_file1
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/global_power_plant_database_v_1_3.zip"

  - id: download_file2
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/enhancing-adaptation-ambition-supplementary-materials.zip"

  - id: upload_files_to_s3
    type: io.kestra.plugin.aws.s3.Upload
    from: |
      [
        "{{ outputs.download_file1.uri }}",
        "{{ outputs.download_file2.uri }}"
      ]
    key: "path/to/files"
    bucket: "{{ inputs.bucket }}"
    region: "{{ secret('AWS_DEFAULT_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"

Upload multiple files to S3 using a Collection

yaml
id: upload_multiple_files_to_s3
namespace: company.team

inputs:
  - id: bucket
    type: STRING
    defaults: my-bucket

tasks:
  - id: download_file1
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/global_power_plant_database_v_1_3.zip"

  - id: download_file2
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/enhancing-adaptation-ambition-supplementary-materials.zip"

  - id: upload_multiple_to_s3
    type: io.kestra.plugin.aws.s3.Upload
    from:
      - "{{ outputs.download_file1.uri }}"
      - "{{ outputs.download_file2.uri }}"
    key: "path/to/files"
    bucket: "{{ inputs.bucket }}"
    region: "{{ secret('AWS_DEFAULT_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"

Upload multiple files to S3 using a JSON map

yaml
id: upload_multiple_files_from_json_map
namespace: company.team

inputs:
  - id: bucket
    type: STRING
    defaults: my-bucket

tasks:
  - id: download_file1
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/global_power_plant_database_v_1_3.zip"

  - id: download_file2
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/enhancing-adaptation-ambition-supplementary-materials.zip"

  - id: upload_files_to_s3
    type: io.kestra.plugin.aws.s3.Upload
    from: |
      [
        "first_key": "{{ outputs.download_file1.uri }}",
        "second_key": "{{ outputs.download_file2.uri }}"
      ]
    key: "path/to/files"
    bucket: "{{ inputs.bucket }}"
    region: "{{ secret('AWS_DEFAULT_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"

Upload multiple files to S3 using a Map

yaml
id: upload_multiple_files_to_s3_from_map
namespace: company.team

inputs:
  - id: bucket
    type: STRING
    defaults: my-bucket

tasks:
  - id: download_file1
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/global_power_plant_database_v_1_3.zip"

  - id: download_file2
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/enhancing-adaptation-ambition-supplementary-materials.zip"

  - id: upload_multiple_to_s3
    type: io.kestra.plugin.aws.s3.Upload
    from:
      firstKey: "{{ outputs.download_file1.uri }}"
      secondKey: "{{ outputs.download_file2.uri }}"
    key: "path/to/files"
    bucket: "{{ inputs.bucket }}"
    region: "{{ secret('AWS_DEFAULT_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"

Properties

bucket *string

The S3 bucket name

from *arraystringobject

Structured data items, either as a map, a list of map, a URI, or a JSON string.

Structured data items can be defined in the following ways:

A single item as a map (a document).
A list of items as a list of maps (a list of documents).
A URI, supported schemes are kestra for internal storage files, file for host local files, and nsfile for namespace files.
A JSON String that will then be serialized either as a single item or a list of items.

key *string

Object key

Full key for single upload or base prefix for multi-file uploads.

accessKeyId string

Access Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

acl string

Canned ACL

bucketKeyEnabled booleanstring

Bucket key enabled

Use S3 Bucket Key when SSE-KMS is selected.

cacheControl string

Cache-Control

checksum string

Checksum value

Must match the selected checksumAlgorithm.

checksumAlgorithm string

Possible Values

CRC32CRC32_CSHA1SHA256CRC64_NVMEUNKNOWN_TO_SDK_VERSION

Checksum algorithm

compatibilityMode booleanstring

Defaultfalse

Compatibility mode

Use default async client for S3-compatible endpoints (limits transfers to ~2GB).

concurrentUploads integerstring

Default50

Max concurrent file uploads

Max upload futures submitted to the TransferManager before waiting for the window to complete. Effective I/O concurrency is also bounded by the HTTP connection pool. Defaults to 50.

contentDisposition string

Content-Disposition

contentEncoding string

Content-Encoding

Applied encodings; informs how to decode to the Content-Type.

contentLanguage string

Content-Language

contentLength integerstring

Content-Length

Explicit length when it cannot be inferred.

contentType string

Content-Type

endpointOverride string

The endpoint with which the SDK should communicate

This property allows you to use a different S3 compatible storage backend.

expectedBucketOwner string

Expected bucket owner

Reject if the bucket is owned by another account.

expires string

Expires

forcePathStyle booleanstring

Force path style access

Must only be used when compatibilityMode is enabled.

metadata object

Metadata

Key/value metadata stored with the object.

objectLockLegalHoldStatus string

Possible Values

ONOFFUNKNOWN_TO_SDK_VERSION

Legal hold

objectLockMode string

Possible Values

GOVERNANCECOMPLIANCEUNKNOWN_TO_SDK_VERSION

Object Lock mode

objectLockRetainUntilDate string

Retain until date

pluginDefaultsRef string

Reference (ref) of the pluginDefaults to apply to this task.

region string

AWS region with which the SDK should communicate

requestPayer string

Sets the value of the RequestPayer property for this object

secretKeyId string

Secret Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

serverSideEncryption string

Possible Values

AES256AWS_FSXAWS_KMSAWS_KMS_DSSEUNKNOWN_TO_SDK_VERSION

Server-side encryption

For example AES256, aws: kms, aws: kms: dsse.

sessionToken string

AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

storageClass string

Possible Values

STANDARDREDUCED_REDUNDANCYSTANDARD_IAONEZONE_IAINTELLIGENT_TIERINGGLACIERDEEP_ARCHIVEOUTPOSTSGLACIER_IRSNOWEXPRESS_ONEZONEFSX_OPENZFSFSX_ONTAPUNKNOWN_TO_SDK_VERSION

Storage class

Defaults to STANDARD if not set.

stsEndpointOverride string

The AWS STS endpoint with which the SDKClient should communicate

stsRoleArn string

AWS STS Role

The Amazon Resource Name (ARN) of the role to assume. If set the task will use the StsAssumeRoleCredentialsProvider. If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

stsRoleExternalId string

AWS STS External Id

A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.

stsRoleSessionDuration string

DefaultPT15M

AWS STS Session duration

The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.

stsRoleSessionName string

AWS STS Session name

This property is only used when an stsRoleArn is defined.

tagging object