Google Cloud Storage (GCS) Upload

Google Cloud Storage (GCS) Upload
Certified
Upload a file to GCS

Reads a file from Kestra internal storage and writes it to a gs:// destination. Supports content metadata settings.

yaml
type: io.kestra.plugin.gcp.gcs.Upload

Examples

Upload a FILE input to GCS

yaml
id: gcp_gcs_upload
namespace: company.team

inputs:
  - id: file
    type: FILE

tasks:
  - id: upload
    type: io.kestra.plugin.gcp.gcs.Upload
    from: "{{ inputs.file }}"
    to: "gs://my_bucket/dir/file.csv"

Download data and upload to Google Cloud Storage

yaml
id: load_to_cloud_storage
namespace: company.team

tasks:
  - id: data
    type: io.kestra.plugin.core.http.Download
    uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv

  - id: cloud_storage
    type: io.kestra.plugin.gcp.gcs.Upload
    from: "{{ outputs.data.uri }}"
    to: gs://kestra-demo/data.csv

Properties

assets

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions

io.kestra.core.models.assets.AssetsDeclaration

assetFailureBehaviorstring

Possible Values

IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

io.kestra.core.models.assets.AssetIdentifier

id*string

Min length1

typestring

outputs

The assets produced as outputs.

io.kestra.plugin.ee.assets.Dataset

id*string

Min length1

Max length150

type*object

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

io.kestra.plugin.ee.assets.File

id*string

Min length1

Max length150

type*object

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

io.kestra.plugin.ee.assets.Table

id*string

Min length1

Max length150

type*object

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

io.kestra.plugin.ee.assets.VM

id*string

Min length1

Max length150

type*object

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

io.kestra.core.models.assets.External

id*string

Min length1

Max length150

type*object

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

io.kestra.core.models.assets.Custom

id*string

Min length1

Max length150

type*string

Min length1

Custom asset type

descriptionstring

displayNamestring

metadataobject

Default{}

namespacestring

Min length1

Max length150

cacheControl string

Cache-Control

contentDisposition string

Content-Disposition

contentEncoding string

Content-Encoding

contentType string

Content-Type

expectedCrc32c string

Expected CRC32C of the source

Optional base64-encoded big-endian CRC32C. When set, the task fails if the source data read from Kestra internal storage does not match this checksum.

expectedMd5 string

Expected MD5 of the source

Optional base64-encoded MD5 digest. When set, the task fails if the source data read from Kestra internal storage does not match this digest. Use this to assert end-to-end integrity against a checksum produced upstream.

from string

Source file URI

Kestra internal storage URI to upload

Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.

impersonatedServiceAccount string

The GCP service account to impersonate

projectId string

The GCP project ID

scopes array

SubTypestring

Default["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used

serviceAccount string

The GCP service account

to string

Destination object URI

gs:// bucket/path for the uploaded object. Special characters and partition brackets are automatically percent-encoded if unencoded.

validateChecksum booleanstring

Defaulttrue

Validate checksum after upload

When true (default), compute MD5 and CRC32C client-side while streaming and compare them against what GCS reports server-side once the upload completes. When false, the implicit transit integrity check is skipped; explicit expectedMd5/expectedCrc32c assertions are still honored when set.

Outputs

crc32c string

Base64-encoded big-endian CRC32C of the uploaded data

md5 string

Base64-encoded MD5 of the uploaded data

uri string

Formaturi

Uploaded object URI

Metrics

file.size counter

Unitbytes

Size of the uploaded file.