upload
Upload a file(s) to a S3 bucket.

Upload a file(s) to a S3 bucket.

Uploads a single or multiple files to an Amazon S3 bucket.

yaml
type: "io.kestra.plugin.aws.s3.upload"

Examples

Upload a FILE input to S3.

yaml
id: aws_s3_upload
namespace: company.team

inputs:
  - id: bucket
    type: STRING
    defaults: my-bucket

  - id: myfile
    type: FILE

tasks:
  - id: upload
    type: io.kestra.plugin.aws.s3.Upload
    region: "{{ secret('AWS_DEFAULT_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
    from: "{{ inputs.myfile }}"
    bucket: "{{ inputs.bucket }}"
    key: "path/to/file"

Download a file and upload it to S3.

yaml
id: upload_file_to_s3
namespace: company.team

inputs:
  - id: bucket
    type: STRING
    defaults: my-bucket

  - id: file_url
    type: STRING
    defaults: https://wri-dataportal-prod.s3.amazonaws.com/manual/global_power_plant_database_v_1_3.zip

tasks:
  - id: download_file
    type: io.kestra.plugin.core.http.Download
    uri: "{{ inputs.file_url }}"

  - id: upload_to_s3
    type: io.kestra.plugin.aws.s3.Upload
    from: "{{ outputs.download_file.uri }}"
    key: powerplant/global_power_plant_database.zip
    bucket: "{{ inputs.bucket }}"
    region: "{{ secret('AWS_DEFAULT_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"

Upload multiple files to S3 using a JSON array.

yaml
id: upload_multiple_files_from_json_array
namespace: company.team

inputs:
  - id: bucket
    type: STRING
    defaults: my-bucket

tasks:
  - id: download_file1
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/global_power_plant_database_v_1_3.zip"

  - id: download_file2
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/enhancing-adaptation-ambition-supplementary-materials.zip"

  - id: upload_files_to_s3
    type: io.kestra.plugin.aws.s3.Upload
    from: |
      [
        "{{ outputs.download_file1.uri }}",
        "{{ outputs.download_file2.uri }}"
      ]
    key: "path/to/files"
    bucket: "{{ inputs.bucket }}"
    region: "{{ secret('AWS_DEFAULT_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"

Upload multiple files to S3 using a Collection.

yaml
id: upload_multiple_files_to_s3
namespace: company.team

inputs:
  - id: bucket
    type: STRING
    defaults: my-bucket

tasks:
  - id: download_file1
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/global_power_plant_database_v_1_3.zip"

  - id: download_file2
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/enhancing-adaptation-ambition-supplementary-materials.zip"

  - id: upload_multiple_to_s3
    type: io.kestra.plugin.aws.s3.Upload
    from:
      - "{{ outputs.download_file1.uri }}"
      - "{{ outputs.download_file2.uri }}"
    key: "path/to/files"
    bucket: "{{ inputs.bucket }}"
    region: "{{ secret('AWS_DEFAULT_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"

Upload multiple files to S3 using a JSON map.

yaml
id: upload_multiple_files_from_json_map
namespace: company.team

inputs:
  - id: bucket
    type: STRING
    defaults: my-bucket

tasks:
  - id: download_file1
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/global_power_plant_database_v_1_3.zip"

  - id: download_file2
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/enhancing-adaptation-ambition-supplementary-materials.zip"

  - id: upload_files_to_s3
    type: io.kestra.plugin.aws.s3.Upload
    from: |
      [
        "first_key": "{{ outputs.download_file1.uri }}",
        "second_key": "{{ outputs.download_file2.uri }}"
      ]
    key: "path/to/files"
    bucket: "{{ inputs.bucket }}"
    region: "{{ secret('AWS_DEFAULT_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"

Upload multiple files to S3 using a Map.

yaml
id: upload_multiple_files_to_s3_from_map
namespace: company.team

inputs:
  - id: bucket
    type: STRING
    defaults: my-bucket

tasks:
  - id: download_file1
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/global_power_plant_database_v_1_3.zip"

  - id: download_file2
    type: io.kestra.plugin.core.http.Download
    uri: "https://wri-dataportal-prod.s3.amazonaws.com/manual/enhancing-adaptation-ambition-supplementary-materials.zip"

  - id: upload_multiple_to_s3
    type: io.kestra.plugin.aws.s3.Upload
    from:
      firstKey: "{{ outputs.download_file1.uri }}"
      secondKey: "{{ outputs.download_file2.uri }}"
    key: "path/to/files"
    bucket: "{{ inputs.bucket }}"
    region: "{{ secret('AWS_DEFAULT_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"

Properties

bucket*string

The S3 bucket name.

from*arraystringobject

Structured data items, either as a map, a list of map, a URI, or a JSON string.

Structured data items can be defined in the following ways:

A single item as a map (a document).
A list of items as a list of maps (a list of documents).
A URI, supported schemes are kestra for internal storage files, file for host local files, and nsfile for namespace files.
A JSON String that will then be serialized either as a single item or a list of items.

key*string

The key where to upload the file

A full key (with filename) or the directory path if from is multiple files

accessKeyIdstring

Access Key Id in order to connect to AWS.

aclstring

The canned ACL to apply to the object

bucketKeyEnabledbooleanstring

Specifies whether Amazon S3 should use an S3 Bucket Key for object encryption with server-side encryption using Key Management Service (KMS) keys (SSE-KMS).

Setting this header to true causes Amazon S3 to use an S3 Bucket Key for object encryption with SSE-KMS.

cacheControlstring

Can be used to specify caching behavior along the request/response chain.

checksumstring

The checksum data integrity check to verify that the data received is the same data that was originally sent.

Must be used in pair with checksumAlgorithm to defined the expect algorithm of these values

checksumAlgorithmstring

Possible Values

CRC32CRC32_CSHA1SHA256CRC64_NVMEUNKNOWN_TO_SDK_VERSION

Indicates the algorithm used to create the checksum for the object when using the SDK

compatibilityModebooleanstring

Defaultfalse

This property will use the AWS S3 DefaultAsyncClient instead of the S3CrtAsyncClient, which maximizes compatibility with S3-compatible services but restricts uploads and downloads to 2GB. For some S3 endpoints such as CloudFlare R2, you may need to set this value to true.

contentDispositionstring

Specifies presentational information for the object

contentEncodingstring

Specifies what content encodings have been applied to the object

And thus, what decoding mechanisms must be applied to obtain the media-type referenced by the Content-Type header field.

contentLanguagestring

The language the content is in

contentLengthintegerstring

The size of the body in bytes

This parameter is useful when the size of the body cannot be determined automatically.

contentTypestring

A standard MIME type that describes the format of the contents.

endpointOverridestring

The endpoint with which the SDK should communicate.

This property allows you to use a different S3 compatible storage backend.

expectedBucketOwnerstring

The account ID of the expected bucket owner

If the bucket is owned by a different account, the request fails with the HTTP status code 403 Forbidden (access denied).

expiresstring

The date and time after which the object is no longer cacheable

forcePathStylebooleanstring

Force path style access.

Must only be used when compatibilityMode is enabled.

metadataobject

SubTypestring

A map of metadata to store with the object in S3.

objectLockLegalHoldStatusstring

Possible Values

ONOFFUNKNOWN_TO_SDK_VERSION

Specifies whether a legal hold will be applied to this object

objectLockModestring

Possible Values

GOVERNANCECOMPLIANCEUNKNOWN_TO_SDK_VERSION

The Object Lock mode that you want to apply to this object

objectLockRetainUntilDatestring

The date and time when you want this object's Object Lock to expire

regionstring

AWS region with which the SDK should communicate.

requestPayerstring

Sets the value of the RequestPayer property for this object.

secretKeyIdstring

Secret Key Id in order to connect to AWS.

serverSideEncryptionstring

Possible Values

AES256AWS_FSXAWS_KMSAWS_KMS_DSSEUNKNOWN_TO_SDK_VERSION

The server-side encryption algorithm used when storing this object in Amazon S3

For example, AES256, aws: kms, aws: kms: dsse

sessionTokenstring

AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource.

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

storageClassstring

Possible Values

STANDARDREDUCED_REDUNDANCYSTANDARD_IAONEZONE_IAINTELLIGENT_TIERINGGLACIERDEEP_ARCHIVEOUTPOSTSGLACIER_IRSNOWEXPRESS_ONEZONEFSX_OPENZFSUNKNOWN_TO_SDK_VERSION

If you don't specify, S3 Standard is the default storage class. Amazon S3 supports other storage classes.

stsEndpointOverridestring

The AWS STS endpoint with which the SDKClient should communicate.

stsRoleArnstring

AWS STS Role.

The Amazon Resource Name (ARN) of the role to assume. If set the task will use the StsAssumeRoleCredentialsProvider. If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

stsRoleExternalIdstring

AWS STS External Id.

A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.

stsRoleSessionDurationstring

DefaultPT15M

Formatduration

AWS STS Session duration.

The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.

stsRoleSessionNamestring

AWS STS Session name.

This property is only used when an stsRoleArn is defined.

taggingobject

SubTypestring

The tag-set for the object

Outputs

bucketstring

The S3 bucket name

The name of the bucket where the file(s) were uploaded

filesobject

Information about uploaded files

A map of file names to their corresponding file information. Returned only for multiple file uploads.

Definitions

io.kestra.plugin.aws.s3.models.FileInfo

contentLengthinteger

The size of the file in bytes

contentTypestring

The MIME type of the file

eTagstring

An ETag is an opaque identifier assigned by a web server to a specific version of a resource found at a URL.

metadataobject

SubTypestring

The metadata of the file

uristring

Formaturi

The URI of the downloaded file in Kestra's storage

versionIdstring

The version ID of the file

keystring

The S3 object key

The key (path) where the file(s) were uploaded in the bucket

versionIdstring

The version of the object.

Metrics

file.countcounter

Unitfiles

Total number of files uploaded.

file.sizecounter

Unitbytes

Total size in bytes of files uploaded.

Core Plugins and tasks

Tasks that provide Kestra's built-in orchestration, I/O, and observability capabilities.

CoreStorageScript

Resend

Plugin Resend for Kestra

Messaging

Ruby

Tasks that run Ruby code.

Script

upload
Upload a file(s) to a S3 bucket.

More Plugins in this Category

Core Plugins and tasks

Resend

Ruby

1.4.0

uploadUpload a file(s) to a S3 bucket.

More Plugins in this Category

Core Plugins and tasks

Resend

Ruby

1.4.0

upload
Upload a file(s) to a S3 bucket.