Compose
This task allows you to concatenate up to 32 files in a given GCS bucket into a single file.
This task allows you to concatenate up to 32 files in a given GCS bucket into a single file.
Compose multiple objects in a GCS bucket into a single object.
This task allows you to concatenate up to 32 files in a given GCS bucket into a single file.
type: "io.kestra.plugin.gcp.gcs.Compose"Examples
Concatenate files in a bucket
id: gcp_gcs_compose
namespace: company.team
tasks:
- id: compose
type: io.kestra.plugin.gcp.gcs.Compose
list:
from: "gs://my_bucket/dir/"
to: "gs://my_bucket/destination/my-compose-file.txt"
Properties
list*Required
io.kestra.plugin.gcp.gcs.Compose-List
The directory to list
BOTHFILESDIRECTORYBOTHDIRECTORYRECURSIVEDIRECTORYThe listing type you want (like directory or recursive)
if DIRECTORY, will only list objects in the specified directory
if RECURSIVE, will list objects in the specified directory recursively
Default value is DIRECTORY
When using RECURSIVE value, be careful to move your files to a location not in the from scope
A regexp to filter on full path
ex:
regExp: .* to match all files
regExp: .*2020-01-0.\\.csv to match files between 01 and 09 of january ending with .csv
allowEmptybooleanstring
falseif true, don't failed if no result
impersonatedServiceAccountstring
The GCP service account to impersonate.
projectIdstring
The GCP project ID.
scopesarray
["https://www.googleapis.com/auth/cloud-platform"]The GCP scopes to be used.
serviceAccountstring
The GCP service account.
tostring
The destination path
Outputs
uristring
uriMetrics
countcounter
filesNumber of files composed.
sizecounter
bytesSize of the composed file.
