Downloads
Downloads
type: "io.kestra.plugin.gcp.gcs.Downloads"
Download multiple files from a GCS bucket.
Examples
Download a list of files and move it to an archive folders
id: gcp_gcs_downloads
namespace: company.team
tasks:
- id: downloads
type: io.kestra.plugin.gcp.gcs.Downloads
from: gs://my-bucket/kestra/files/
action: MOVE
moveDirectory: gs://my-bucket/kestra/archive/
Properties
action
- Type: string
- Dynamic: ✔️
- Required: ✔️
- Possible Values:
MOVE
DELETE
NONE
The action to perform on the retrieved files. If using 'NONE' make sure to handle the files inside your flow to avoid infinite triggering.
from
- Type: string
- Dynamic: ✔️
- Required: ✔️
The directory to list
allVersions
- Type: boolean
- Dynamic: ✔️
- Required: ❌
If set to true
, lists all versions of a blob. The default is false
.
impersonatedServiceAccount
- Type: string
- Dynamic: ✔️
- Required: ❌
The GCP service account to impersonate.
listingType
- Type: string
- Dynamic: ❌
- Required: ❌
- Default:
DIRECTORY
- Possible Values:
RECURSIVE
DIRECTORY
The listing type you want (like directory or recursive)
if DIRECTORY, will only list objects in the specified directory if RECURSIVE, will list objects in the specified directory recursively Default value is DIRECTORY When using RECURSIVE value, be careful to move your files to a location not in the
from
scope
moveDirectory
- Type: string
- Dynamic: ✔️
- Required: ❌
The destination directory for MOVE
action.
projectId
- Type: string
- Dynamic: ✔️
- Required: ❌
The GCP project ID.
regExp
- Type: string
- Dynamic: ✔️
- Required: ❌
A regexp to filter on full path
ex:
regExp: .*
to match all filesregExp: .*2020-01-0.\\.csv
to match files between 01 and 09 of january ending with.csv
scopes
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
- Default:
[https://www.googleapis.com/auth/cloud-platform]
The GCP scopes to be used.
serviceAccount
- Type: string
- Dynamic: ✔️
- Required: ❌
The GCP service account.
Outputs
blobs
- Type: array
- SubType: Blob
- Required: ❌
The bucket of the downloaded file
outputFiles
- Type: object
- SubType: string
- Required: ❌
The downloaded files as a map of from/to URIs.
Definitions
io.kestra.plugin.gcp.gcs.models.Blob
Properties
bucket
- Type: string
- Dynamic: ❓
- Required: ❓
cacheControl
- Type: string
- Dynamic: ❓
- Required: ❓
componentCount
- Type: integer
- Dynamic: ❓
- Required: ❓
contentDisposition
- Type: string
- Dynamic: ❓
- Required: ❓
contentEncoding
- Type: string
- Dynamic: ❓
- Required: ❓
contentLanguage
- Type: string
- Dynamic: ❓
- Required: ❓
contentType
- Type: string
- Dynamic: ❓
- Required: ❓
crc32c
- Type: string
- Dynamic: ❓
- Required: ❓
createTime
- Type: string
- Dynamic: ❓
- Required: ❓
- Format:
date-time
customTime
- Type: string
- Dynamic: ❓
- Required: ❓
- Format:
date-time
deleteTime
- Type: string
- Dynamic: ❓
- Required: ❓
- Format:
date-time
etag
- Type: string
- Dynamic: ❓
- Required: ❓
eventBasedHold
- Type: boolean
- Dynamic: ❓
- Required: ❓
generatedId
- Type: string
- Dynamic: ❓
- Required: ❓
isDirectory
- Type: boolean
- Dynamic: ❓
- Required: ❓
kmsKeyName
- Type: string
- Dynamic: ❓
- Required: ❓
md5
- Type: string
- Dynamic: ❓
- Required: ❓
mediaLink
- Type: string
- Dynamic: ❓
- Required: ❓
metaGeneration
- Type: integer
- Dynamic: ❓
- Required: ❓
metadata
- Type: object
- SubType: string
- Dynamic: ❓
- Required: ❓
name
- Type: string
- Dynamic: ❓
- Required: ❓
retentionExpirationTime
- Type: integer
- Dynamic: ❓
- Required: ❓
selfLink
- Type: string
- Dynamic: ❓
- Required: ❓
size
- Type: integer
- Dynamic: ❓
- Required: ❓
temporaryHold
- Type: boolean
- Dynamic: ❓
- Required: ❓
timeStorageClassUpdated
- Type: string
- Dynamic: ❓
- Required: ❓
- Format:
date-time
updateTime
- Type: string
- Dynamic: ❓
- Required: ❓
- Format:
date-time
uri
- Type: string
- Dynamic: ❓
- Required: ❓
- Format:
uri
Was this page helpful?