Google Cloud Compose
CertifiedCompose multiple GCS objects into one
Google Cloud Compose
Compose multiple GCS objects into one
Concatenates up to 32 source objects into a single destination object in GCS.
type: io.kestra.plugin.gcp.gcs.ComposeExamples
Concatenate files in a bucket
id: gcp_gcs_compose
namespace: company.team
tasks:
- id: compose
type: io.kestra.plugin.gcp.gcs.Compose
list:
from: "gs://my_bucket/dir/"
to: "gs://my_bucket/destination/my-compose-file.txt"
Properties
list *Required
Source listing
Listing definition used to select source objects (files only)
io.kestra.plugin.gcp.gcs.Compose-List
Source prefix
gs:// path to list
BOTHFILESDIRECTORYBOTHDIRECTORYRECURSIVEDIRECTORYListing type
DIRECTORY lists only immediate children; RECURSIVE traverses all descendants. Default DIRECTORY.
Regex filter
Optional regex applied to full object path (e.g., .*2020-01-0.\.csv)
allowEmpty booleanstring
falseAllow empty source
If true, succeed with empty sources; otherwise fail when no files are found
impersonatedServiceAccount string
The GCP service account to impersonate
pluginDefaultsRef Non-dynamicstring
Reference (ref) of the pluginDefaults to apply to this task.
projectId string
The GCP project ID
scopes array
["https://www.googleapis.com/auth/cloud-platform"]The GCP scopes to be used
serviceAccount string
The GCP service account
to string
Destination object URI
gs:// path for the composed object
Outputs
uri string
uriThe URI of the composed object on Cloud Storage
Metrics
count counter
filesNumber of files composed.
size counter
bytesSize of the composed file.
