Google Cloud StorageWrite
CertifiedStream a file to BigQuery via Storage Write API
Google Cloud StorageWrite
Stream a file to BigQuery via Storage Write API
Reads a file from Kestra internal storage and writes rows with the BigQuery Storage Write API. Supports DEFAULT, COMMITTED, or PENDING streams (commit required for PENDING). Requires the destination table to exist.
type: io.kestra.plugin.gcp.bigquery.StorageWriteExamples
id: gcp_bq_storage_write
namespace: company.team
tasks:
- id: read_data
type: io.kestra.plugin.core.http.Download
uri: https://dummyjson.com/products/1
- id: storage_write
type: io.kestra.plugin.gcp.bigquery.StorageWrite
from: "{{ outputs.read_data.uri }}"
destinationTable: "my-project.my_dataset.my_table"
writeStreamType: DEFAULT
Properties
destinationTable *string
The table where to load data
The table must be created before.
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150bufferSize integerstring
1000The number of records to send on each query
from string
The fully-qualified URIs that point to source data
impersonatedServiceAccount string
The GCP service account to impersonate
location string
The geographic location where the dataset should reside
This property is experimental and might be subject to change or removed.
See Dataset Location
projectId string
The GCP project ID
scopes array
["https://www.googleapis.com/auth/cloud-platform"]The GCP scopes to be used
serviceAccount string
The GCP service account
writeStreamType string
DEFAULTDEFAULTCOMMITTEDPENDINGThe type of write stream to use
Outputs
commitTime string
date-timeCommit time reported by BigQuery, only on PENDING writeStreamType
rows integer
Rows count
rowsCount integer
Rows count reported by BigQuery, only on PENDING writeStreamType
Metrics
rows counter
Rows count.
rows_count counter
Rows count reported by BigQuery, only on PENDING writeStreamType.
