Azure Blob Storage CopyList
CertifiedCopy multiple Azure Blobs
Azure Blob Storage CopyList
Copy multiple Azure Blobs
Lists blobs matching the prefix, regexp and filter, and copies them server-side to a destination container and prefix. Authentication must provide an account key to sign a read SAS on each source blob: use connectionString containing AccountKey=, or both sharedKeyAccountName and sharedKeyAccountAccessKey. sasToken, a connection string carrying only a SAS, and default credentials are not supported. When copying inside the same container, to.prefix must differ from prefix and must not be nested under it unless a delimiter is set.
type: io.kestra.plugin.azure.storage.blob.CopyListExamples
Batch copy blobs from one prefix to another container
id: azure_storage_blob_copy_list
namespace: company.team
tasks:
- id: copy_blobs
type: io.kestra.plugin.azure.storage.blob.CopyList
endpoint: "https://yourblob.blob.core.windows.net"
connectionString: "{{ secret('AZURE_STORAGE_CONNECTION_STRING') }}"
container: "staging"
prefix: "daily-export/"
to:
container: "production"
prefix: "daily-export/"
delete: false
Properties
container *string
The blob container.
endpoint *string
Service endpoint URL
Base HTTPS endpoint of the target service.
to *
Destination configuration (container, prefix)
io.kestra.plugin.azure.storage.blob.models.CopyListDestination
The destination container
The destination prefix
Replaces the source prefix in each copied blob name. When no source prefix is set, it is prepended to the blob name.
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150concurrent integerstring
Parallel copies
Maximum number of concurrent copy operations, between 2 and 64. Blobs are copied sequentially when not set.
connectionString string
Connection string of the Storage Account.
delete booleanstring
falseDelete source blobs after copy
If true, deletes each source blob after it has been successfully copied (move operation).
delimiter string
The delimiter for blob hierarchy, "/" for hierarchy based on directories.
filter string
FILESFILESDIRECTORYBOTHThe filter for files or directories.
prefix string
Limits the response to keys that begin with the specified prefix.
regexp string
A regular expression to filter on the full key.
ex:
regExp: .* to match all files
regExp: .*2020-01-0.\\.csv to match files between 01 and 09 of january ending with .csv
sasToken string
The SAS token to use for authenticating requests.
This string should only be the query parameters (with or without a leading '?') and not a full URL.
Outputs
count integer
0Number of blobs copied
size integer
0Total size in bytes of copied blobs
Metrics
azure.blob.copied.count counter
blobsThe total number of blobs copied.
azure.blob.copied.size counter
bytesThe total size in bytes of blobs copied.
