AWS S3 CopyList

AWS S3 CopyList

Certified

Copy multiple S3 objects

Lists the objects matching prefix, regexp and filter, then copies each one server-side under to.prefix, optionally in parallel and deleting the source after the copy (a move). A key keeps its path relative to prefix, or its full key when prefix is not set: with prefix: a/ and to.prefix: b/, a/x/y.csv is copied to b/x/y.csv. Only the first maxKeys (default 1000) matching objects are handled per run. Server-side copy is limited to objects up to 5 GB, use the Copy task for larger objects.

yaml
type: io.kestra.plugin.aws.s3.CopyList

Copy all objects under a prefix to another bucket

yaml
id: aws_s3_copy_list
namespace: company.team

tasks:
  - id: copy_prefix
    type: io.kestra.plugin.aws.s3.CopyList
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
    region: "eu-central-1"
    bucket: "source-bucket"
    prefix: "daily-export/"
    to:
      bucket: "dest-bucket"
      prefix: "backup/"

Move objects matching a regex within the same bucket (delete source after copy)

yaml
id: aws_s3_move_list
namespace: company.team

tasks:
  - id: move_logs
    type: io.kestra.plugin.aws.s3.CopyList
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_KEY_ID') }}"
    region: "eu-central-1"
    bucket: "company-logs"
    prefix: "inbound/"
    regexp: ".*\\.json"
    to:
      prefix: "processed/"
    delete: true
    concurrent: 4
Properties

The S3 bucket where to download the file

Destination bucket and prefix

Definitions
bucketstring

The destination bucket

Defaults to the source bucket. In the same bucket, to.prefix must be outside prefix unless a delimiter is set.

kmsKeyIdstring

KMS key ARN or ID, when serverSideEncryption is AWS_KMS

prefixstring

The destination prefix

Each key keeps its path relative to prefix under it. Defaults to the bucket root.

serverSideEncryptionstring
Possible Values
NONEAES256AWS_KMS

Server-side encryption for the copied objects

Access Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions
assetFailureBehaviorstring
Possible Values
IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

id*string
Min length1
typestring
outputs

The assets produced as outputs.

id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*string
Min length1

Custom asset type

descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150

Enable compatibility mode

Use it to connect to S3 bucket with S3 compatible services that don't support the new transport client.

Parallel copies

Copies run one at a time when unset, otherwise between 2 and 64 at once.

Defaultfalse

Delete source objects after copy

When true, each source object is deleted once its copy succeeded, which makes the task a move.

A delimiter is a character you use to group keys

The EncodingType property for this object

The endpoint with which the SDK should communicate

This property allows you to use a different S3 compatible storage backend.

Defaultfalse

Error if empty

When true, the task fails when no object matches the filters.

The account ID of the expected bucket owner

If the bucket is owned by a different account, the request fails with the HTTP status code 403 Forbidden (access denied).

DefaultBOTH
Possible Values
FILESDIRECTORYBOTH

The type of objects to filter: files, directory, or both

Force path style access

Must only be used when compatibilityMode is enabled.

Marker is where you want Amazon S3 to start listing from

Amazon S3 starts listing after this specified key. Marker can be any key in the bucket.

Default1000

Maximum number of objects returned across all paginated S3 calls

Total upper bound on the objects returned, counted after regexp and filter. Default 1000, must be 0 or more, and 0 returns no objects. When regexp is unset and filter is BOTH, it also sets the per-page size sent to S3, which AWS caps at 1000. The listing pages until this total is reached or the bucket is exhausted, so a regexp or filter that matches few keys can read every key under prefix. Set prefix to narrow it.

Limits the response to keys that begin with the specified prefix

A regexp to filter on full key

ex: regExp: .* to match all files regExp: .*2020-01-0.\\.csv to match files between 01 and 09 of january ending with .csv

AWS region with which the SDK should communicate

Sets the value of the RequestPayer property for this object

Secret Key Id in order to connect to AWS

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

AWS session token, retrieved from an AWS token service, used for authenticating that this user has received temporary permissions to access a given resource

If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

The AWS STS endpoint with which the SDKClient should communicate

AWS STS Role

The Amazon Resource Name (ARN) of the role to assume. If set the task will use the StsAssumeRoleCredentialsProvider. If no credentials are defined, we will use the default credentials provider chain to fetch credentials.

AWS STS External Id

A unique identifier that might be required when you assume a role in another account. This property is only used when an stsRoleArn is defined.

DefaultPT15M

AWS STS Session duration

The duration of the role session (default: 15 minutes, i.e., PT15M). This property is only used when an stsRoleArn is defined.

AWS STS Session name

This property is only used when an stsRoleArn is defined.

Default0

The number of objects copied

Default0

The total size in bytes of the objects copied

Unitobjects

The total number of objects copied.

Unitbytes

The total size in bytes of objects copied.

More Plugins in this Category

Firecrawl
Firecrawl

Tasks that scrape and crawl websites into clean, LLM-ready content with Firecrawl.

Data

2 tasks

Canonical (EE)
Canonical
Enterprise

Tasks to interact with Canonical services, enabling automation and infrastructure management through API-driven workflows.

CloudInfrastructure

5 tasks