Google Cloud Storage (GCS) Compose

Google Cloud Storage (GCS) Compose

Certified

Compose multiple GCS objects into one

Concatenates up to 32 source objects into a single destination object in GCS.

yaml
type: io.kestra.plugin.gcp.gcs.Compose

Concatenate files in a bucket

yaml
id: gcp_gcs_compose
namespace: company.team

tasks:
  - id: compose
    type: io.kestra.plugin.gcp.gcs.Compose
    list:
      from: "gs://my_bucket/dir/"
    to: "gs://my_bucket/destination/my-compose-file.txt"
Properties

Source listing

Listing definition used to select source objects (files only)

Definitions
from*string

Source prefix

gs:// path to list

filterstring
DefaultBOTH
Possible Values
FILESDIRECTORYBOTH
listingTypestring
DefaultDIRECTORY
Possible Values
RECURSIVEDIRECTORY

Listing type

DIRECTORY lists only immediate children; RECURSIVE traverses all descendants. Default DIRECTORY.

regExpstring

Regex filter

Optional regex applied to full object path (e.g., .*2020-01-0.\.csv). Matches against raw URI or encoded URI.

Defaultfalse

Allow empty source

If true, succeed with empty sources; otherwise fail when no files are found

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions
assetFailureBehaviorstring
Possible Values
IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

id*string
Min length1
typestring
outputs

The assets produced as outputs.

id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*string
Min length1

Custom asset type

descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150

The GCP service account to impersonate

The GCP project ID

SubTypestring
Default["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used

The GCP service account

Destination object URI

gs:// path for the composed object. Special characters and partition brackets are automatically percent-encoded if unencoded.

Formaturi

The URI of the composed object on Cloud Storage

Unitfiles

Number of files composed.

Unitbytes

Size of the composed file.

More Plugins in this Category

Firecrawl
Firecrawl

Tasks that scrape and crawl websites into clean, LLM-ready content with Firecrawl.

Data

2 tasks

Canonical (EE)
Canonical
Enterprise

Tasks to interact with Canonical services, enabling automation and infrastructure management through API-driven workflows.

CloudInfrastructure

5 tasks