Google Cloud Storage (GCS) Downloads
Certified
Download multiple GCS objects
Google Cloud Storage (GCS) Downloads
Download multiple GCS objects
Lists objects under a prefix (with optional regex and versioning) then downloads them to Kestra storage. Can MOVE (copy then delete) or DELETE source objects after download.
type: io.kestra.plugin.gcp.gcs.DownloadsExamples
Download a list of files and move it to an archive folders
id: gcp_gcs_downloads
namespace: company.team
tasks:
- id: downloads
type: io.kestra.plugin.gcp.gcs.Downloads
from: gs://my-bucket/kestra/files/
action: MOVE
moveDirectory: gs://my-bucket/kestra/archive/
Download a list of files with end-to-end checksum validation
id: gcp_gcs_downloads_validated
namespace: company.team
tasks:
- id: downloads
type: io.kestra.plugin.gcp.gcs.Downloads
from: gs://my-bucket/kestra/files/
action: DELETE
validateChecksum: true
Installation
Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command:
kestractl plugins get io.kestra.plugin:plugin-gcp --compatible-for $KESTRA_VERSION For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.
Properties
from *string
Source prefix
gs:// bucket path to list and download from
action string
NONEMOVEDELETENONEPost-download action
NONE (default), DELETE, or MOVE (copy to moveDirectory then delete source)
allVersions booleanstring
If set to true, lists all versions of a blob. The default is false
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150impersonatedServiceAccount string
The GCP service account to impersonate
listingType string
DIRECTORYRECURSIVEDIRECTORYListing type
DIRECTORY lists only immediate children; RECURSIVE traverses all descendants. Default DIRECTORY.
maxFiles integerstring
25Max files
Maximum number of files to list and download
moveDirectory string
Move destination
Required when action is MOVE; gs:// prefix for moved objects
projectId string
The GCP project ID
regExp string
Regex filter
Optional regex applied to object names
scopes array
["https://www.googleapis.com/auth/cloud-platform"]The GCP scopes to be used
serviceAccount string
The GCP service account
When unset or rendered to an empty value, Application Default Credentials are used (e.g. GOOGLE_APPLICATION_CREDENTIALS or the runtime's attached identity).
validateChecksum booleanstring
falseValidate each downloaded file against the GCS checksum
When true, every downloaded file is streamed through a CRC32C hasher (or MD5 if the object has no CRC32C) and compared to the checksum reported by Google Cloud Storage. If any file mismatches, the task fails. Used for end-to-end integrity, not for security/authentication.
workloadIdentityAudience string
Audience of the Workload Identity Federation provider
The full resource name of the workload identity pool provider, //iam.googleapis.com/projects/<number>/locations/global/workloadIdentityPools/<pool>/providers/<provider>.
Required with workloadIdentityToken, and must match the aud claim of the token.
workloadIdentityToken string
OIDC token for Workload Identity Federation
When set together with workloadIdentityAudience, the token is exchanged at Google STS for federated credentials and no service account key is needed: Google verifies it against the identity provider that issued it.
Use {{ oidcToken(audience='…') }} to have Kestra Enterprise mint one for the running execution.
Combine with impersonatedServiceAccount to act as a service account the pool may impersonate, and set projectId explicitly since no key file carries it.
Cannot be combined with serviceAccount.
Outputs
blobs array
Downloaded objects
io.kestra.plugin.gcp.gcs.models.Blob
date-timedate-timedate-timedate-timedate-timeurioutputFiles object
Downloaded file map
Map of object names to Kestra storage URIs
Metrics
checksum.validated counter
Number of files whose checksum was successfully validated after download.