Google Cloud Load
CertifiedLoad an internal file into BigQuery
Google Cloud Load
Load an internal file into BigQuery
Streams a file from Kestra internal storage to a BigQuery load job. Fails on empty files by default (failedOnEmpty true).
type: io.kestra.plugin.gcp.bigquery.LoadExamples
Load an csv file from an input file
id: gcp_bq_load
namespace: company.team
tasks:
- id: load
type: io.kestra.plugin.gcp.bigquery.Load
from: "{{ inputs.file }}"
destinationTable: "my-project.my_dataset.my_table"
format: CSV
csvOptions:
fieldDelimiter: ";"
Properties
autodetect booleanstring
Autodetect source options
Experimental. Lets BigQuery infer schema/options for CSV or JSON sources.
avroOptions Non-dynamic
Avro parsing options
io.kestra.plugin.gcp.bigquery.AbstractLoad-AvroOptions
If format is set to AVRO, you can interpret logical types into their corresponding types (such as TIMESTAMP) instead of only using their raw types (such as INTEGER)
The value may be null.
clusteringFields array
Clustering fields
createDisposition string
CREATE_IF_NEEDEDCREATE_NEVERCreate disposition
Whether the job may create the destination table
csvOptions Non-dynamic
CSV parsing options
io.kestra.plugin.gcp.bigquery.AbstractLoad-CsvOptions
Whether BigQuery should accept rows that are missing trailing optional columns
If true, BigQuery treats missing trailing columns as null values. If {@code false}, records with missing trailing columns are treated as bad records, and if there are too many bad records, an invalid error is returned in the job result. By default, rows with missing trailing columns are considered bad records.
Whether BigQuery should allow quoted data sections that contain newline characters in a CSV file
By default quoted newline are not allowed.
The character encoding of the data
The supported values are UTF-8 or ISO-8859-1. The default value is UTF-8. BigQuery decodes the data after the raw, binary data has been split using the values set in {@link #setQuote(String)} and {@link #setFieldDelimiter(String)}.
The separator for fields in a CSV file
BigQuery converts the string to ISO-8859-1 encoding, and then uses the first byte of the encoded string to split the data in its raw, binary state. BigQuery also supports the escape sequence "\t" to specify a tab separator. The default value is a comma (',').
The value that is used to quote data sections in a CSV file
BigQuery converts the string to ISO-8859-1 encoding, and then uses the first byte of the encoded string to split the data in its raw, binary state. The default value is a double-quote ('"'). If your data does not contain quoted sections, set the property value to an empty string. If your data contains quoted newline characters, you must also set {@link #setAllowQuotedNewLines(boolean)} property to {@code true}.
The number of rows at the top of a CSV file that BigQuery will skip when reading the data
The default value is 0. This property is useful if you have header rows in the file that should be skipped.
destinationTable string
Destination table
Target table for the load job; creation depends on createDisposition
failedOnEmpty booleanstring
trueFail on empty file
If true (default), raises an error when the input file has zero bytes
format Non-dynamicstring
CSVJSONAVROPARQUETORCSource format
from string
Source file URI
Kestra internal storage URI for the file to load
Pebble expression referencing an Internal Storage URI e.g. {{ outputs.mytask.uri }}.
ignoreUnknownValues booleanstring
Ignore unknown values
If true, extra columns are skipped; if false, extra columns count as bad records. Default is false.
impersonatedServiceAccount string
The GCP service account to impersonate
location string
Dataset location
Optional BigQuery location for created or targeted resources. Experimental and may change; see BigQuery dataset location documentation.
maxBadRecords integerstring
Max bad records
Number of bad records allowed before the job fails; default 0
pluginDefaultsRef Non-dynamicstring
Reference (ref) of the pluginDefaults to apply to this task.
projectId string
The GCP project ID
retryAuto Non-dynamic
Automatic BigQuery retry policy
Optional custom retry policy for retryable BigQuery errors. If unset, uses an exponential backoff starting at 5s, up to 15m, max 10 attempts.
Constant retry
Retry with a fixed delay between attempts.
durationRETRY_FAILED_TASKRETRY_FAILED_TASKCREATE_NEW_EXECUTION>= 1durationfalseExponential retry
Retry with exponentially increasing delays between attempts.
durationdurationRETRY_FAILED_TASKRETRY_FAILED_TASKCREATE_NEW_EXECUTION>= 1durationfalseRandom retry
Retry with a random delay within a configurable range between attempts.
durationdurationRETRY_FAILED_TASKRETRY_FAILED_TASKCREATE_NEW_EXECUTION>= 1durationfalseretryMessages array
["due to concurrent update","Retrying the job may solve the problem","Retrying may solve the problem"]Retry message substrings
Case-insensitive substrings that, if found in the error message, trigger an automatic retry
retryReasons array
["rateLimitExceeded","jobBackendError","backendError","internalError","jobInternalError"]Retry reasons
BigQuery error reasons that trigger an automatic retry; evaluated against error reason strings
schema object
Destination schema
Table schema definition; may be omitted when loading into an existing table or supported backup formats
schemaUpdateOptions array
ALLOW_FIELD_ADDITIONALLOW_FIELD_RELAXATIONSchema update options
Experimental. Applies only with WRITE_APPEND or partitioned WRITE_TRUNCATE destinations.
scopes array
["https://www.googleapis.com/auth/cloud-platform"]The GCP scopes to be used
serviceAccount string
The GCP service account
timePartitioningField string
Time partitioning field
timePartitioningType string
DAYDAYHOURMONTHYEARTime partitioning type
Defaults to DAY when partitioning is configured
writeDisposition string
WRITE_TRUNCATEWRITE_TRUNCATE_DATAWRITE_APPENDWRITE_EMPTYWrite disposition
Action if destination exists (e.g., WRITE_APPEND, WRITE_TRUNCATE)
Outputs
destinationTable string
Destination table
jobId string
The job id
rows integer
Output rows count
Metrics
bad.records counter
recordsthe number of bad records reported in a job.
duration timer
The time it took for the task to run.
input.bytes counter
bytesThe number of bytes of source data in a load job.
input.files counter
filesThe number of source files in a load job.
output.bytes counter
bytesThe size of the data loaded by a load job so far, in bytes.
output.rows counter
recordsThe number of rows loaded by a load job so far.
