PluginsGoogle CloudTasksGcsCompose

Compose​Compose

Compose Compose

yaml
type: "io.kestra.plugin.gcp.gcs.Compose"

List file on a GCS bucket.

Examples

Concat files in a bucket

yaml
id: "compose"
type: "io.kestra.plugin.gcp.gcs.Compose"
list: 
  from: "gs://my_bucket/dir/"
to: "gs://my_bucket/destination/my-compose-file.txt"

Properties

list

The directory to list

allowEmpty

  • Type: boolean
  • Dynamic: ❌
  • Required: ❌
  • Default: false

if true, don't failed if no result

projectId

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The GCP project ID.

scopes

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: [https://www.googleapis.com/auth/cloud-platform]

The GCP scopes to be used.

serviceAccount

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The GCP service account key.

to

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The destination path

Outputs

uri

  • Type: string
  • Dynamic: ❓
  • Required: ❌
  • Format: uri

Definitions

io.kestra.plugin.gcp.gcs.Compose-List

Properties

from
  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The directory to list

filter
  • Type: string
  • Dynamic: ❓
  • Required: ❌
  • Default: BOTH
  • Possible Values:
    • FILES
    • DIRECTORY
    • BOTH
listingType
  • Type: string
  • Dynamic: ❌
  • Required: ❌
  • Default: DIRECTORY
  • Possible Values:
    • RECURSIVE
    • DIRECTORY

The listing type you want (like directory or recursive)

if DIRECTORY, will only list objects in the specified directory if RECURSIVE, will list objects in the specified directory recursively Default value is DIRECTORY When using RECURSIVE value, be carefull to move your files to a location not in the from scope

regExp
  • Type: string
  • Dynamic: ✔️
  • Required: ❌

A regexp to filter on full path

ex: regExp: .* to match all files regExp: .*2020-01-0.\\.csv to match files between 01 and 09 of january ending with .csv