StorageWrite
For more details, check out the BigQuery Storage API.
For more details, check out the BigQuery Storage API.
Load a Kestra internal storage file into BigQuery using the BigQuery Storage API.
For more details, check out the BigQuery Storage API.
type: "io.kestra.plugin.gcp.bigquery.StorageWrite"Examples
id: gcp_bq_storage_write
namespace: company.team
tasks:
- id: read_data
type: io.kestra.plugin.core.http.Download
uri: https://dummyjson.com/products/1
- id: storage_write
type: io.kestra.plugin.gcp.bigquery.StorageWrite
from: "{{ outputs.read_data.uri }}"
destinationTable: "my_project.my_dataset.my_table"
writeStreamType: DEFAULT
Properties
destinationTable*Requiredstring
The table where to load data
The table must be created before.
bufferSizeintegerstring
1000The number of records to send on each query
fromstring
The fully-qualified URIs that point to source data
impersonatedServiceAccountstring
The GCP service account to impersonate.
locationstring
The geographic location where the dataset should reside
This property is experimental and might be subject to change or removed.
See Dataset Location
projectIdstring
The GCP project ID.
scopesarray
["https://www.googleapis.com/auth/cloud-platform"]The GCP scopes to be used.
serviceAccountstring
The GCP service account.
writeStreamTypestring
DEFAULTDEFAULTCOMMITTEDPENDINGThe type of write stream to use
Outputs
commitTimestring
date-timeCommit time reported by BigQuery, only on PENDING writeStreamType
rowsinteger
Rows count
rowsCountinteger
Rows count reported by BigQuery, only on PENDING writeStreamType
Metrics
rowscounter
Rows count.
rows_countcounter
Rows count reported by BigQuery, only on PENDING writeStreamType.
