StorageWrite

type: "io.kestra.plugin.gcp.bigquery.StorageWrite"

Load an kestra internal storage file on bigquery using BigQuery Storage API (opens new window)

Examples

id: "storage_write"
type: "io.kestra.plugin.gcp.bigquery.StorageWrite"
from: "{{ outputs.read.uri }}"
destinationTable: "my_project.my_dataset.my_table"
writeStreamType: DEFAULT

Properties

bufferSize

• Type: integer
• Dynamic: ❌
• Required: ✔️
• Default: 1000

The number of records to send on each query

destinationTable

• Type: string
• Dynamic: ✔️
• Required: ✔️
• Min length: 1

The table where to load data

The table must be created before.

from

• Type: string
• Dynamic: ✔️
• Required: ❌

The fully-qualified URIs that point to source data

location

• Type: string
• Dynamic: ✔️
• Required: ❌

The geographic location where the dataset should reside

This property is experimental and might be subject to change or removed.

See Dataset Location

projectId

• Type: string
• Dynamic: ✔️
• Required: ❌

The GCP project id

scopes

• Type: array
• SubType: string
• Dynamic: ✔️
• Required: ❌
• Default: [https://www.googleapis.com/auth/cloud-platform]

The GCP scopes to used

serviceAccount

• Type: string
• Dynamic: ✔️
• Required: ❌

The GCP service account key

writeStreamType

• Type: string

• Dynamic: ❌

• Required: ✔️

• Default: DEFAULT

• Possible Values:

  • DEFAULT
  • COMMITTED
  • PENDING

The type of write stream to use

Outputs

commitTime

• Type: string

Commit time reported by BigQuery, only on PENDING writeStreamType

rows

• Type: integer

Rows count

rowsCount

• Type: integer

Rows count reported by BigQuery, only on PENDING writeStreamType