Compose
yaml
type: "io.kestra.plugin.gcp.gcs.Compose"
List file on a GCS bucket.
Examples
Concat files in a bucket
yaml
id: "compose"
type: "io.kestra.plugin.gcp.gcs.Compose"
list:
from: "gs://my_bucket/dir/"
to: "gs://my_bucket/destination/my-compose-file.txt"
Properties
list
- Type: List
- Dynamic: ✔️
- Required: ✔️
The directory to list
allowEmpty
- Type: boolean
- Dynamic: ❌
- Required: ❌
- Default:
false
if
true
, don't failed if no result
projectId
- Type: string
- Dynamic: ✔️
- Required: ❌
The GCP project id
scopes
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
- Default:
[https://www.googleapis.com/auth/cloud-platform]
The GCP scopes to used
serviceAccount
- Type: string
- Dynamic: ✔️
- Required: ❌
The GCP service account key
to
- Type: string
- Dynamic: ✔️
- Required: ❌
The destination path
Outputs
uri
- Type: string
Definitions
List
from
- Type: string
- Dynamic: ✔️
- Required: ✔️
The directory to list
filter
- Type: string
- Dynamic: ❓
- Required: ❌
- Default:
BOTH
- Possible Values:
FILES
DIRECTORY
BOTH
listingType
- Type: string
- Dynamic: ❌
- Required: ❌
- Default:
DIRECTORY
- Possible Values:
RECURSIVE
DIRECTORY
The listing type you want (like directory or recursive)
if DIRECTORY, will only list objects in the specified directory
if RECURSIVE, will list objects in the specified directory recursively
Default value is DIRECTORY
When using RECURSIVE value, be carefull to move your files to a location not in the from
scope
regExp
- Type: string
- Dynamic: ✔️
- Required: ❌
A regexp to filter on full path
ex:
regExp: .*
to match all files
regExp: .*2020-01-0.\\.csv
to match files between 01 and 09 of january ending with .csv