Google Cloud Compose
CertifiedCompose multiple GCS objects into one
Google Cloud Compose
Compose multiple GCS objects into one
Concatenates up to 32 source objects into a single destination object in GCS.
type: io.kestra.plugin.gcp.gcs.ComposeExamples
Concatenate files in a bucket
id: gcp_gcs_compose
namespace: company.team
tasks:
- id: compose
type: io.kestra.plugin.gcp.gcs.Compose
list:
from: "gs://my_bucket/dir/"
to: "gs://my_bucket/destination/my-compose-file.txt"
Properties
list *
Source listing
Listing definition used to select source objects (files only)
io.kestra.plugin.gcp.gcs.Compose-List
Source prefix
gs:// path to list
BOTHFILESDIRECTORYBOTHDIRECTORYRECURSIVEDIRECTORYListing type
DIRECTORY lists only immediate children; RECURSIVE traverses all descendants. Default DIRECTORY.
Regex filter
Optional regex applied to full object path (e.g., .*2020-01-0.\.csv)
allowEmpty booleanstring
falseAllow empty source
If true, succeed with empty sources; otherwise fail when no files are found
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150impersonatedServiceAccount string
The GCP service account to impersonate
projectId string
The GCP project ID
scopes array
["https://www.googleapis.com/auth/cloud-platform"]The GCP scopes to be used
serviceAccount string
The GCP service account
to string
Destination object URI
gs:// path for the composed object
Outputs
uri string
uriThe URI of the composed object on Cloud Storage
Metrics
count counter
filesNumber of files composed.
size counter
bytesSize of the composed file.
