Load data from local file to BigQuery
type: "io.kestra.plugin.gcp.bigquery.Load"Load an csv file from an input file
id: gcp_bq_load
namespace: company.team
tasks:
- id: load
type: io.kestra.plugin.gcp.bigquery.Load
from: "{{ inputs.file }}"
destinationTable: "my_project.my_dataset.my_table"
format: CSV
csvOptions:
fieldDelimiter: ";"
YESExperimental Automatic inference of the options and schema for CSV and JSON sources.
NOAvro parsing options.
YESThe clustering specification for the destination table.
YESCREATE_IF_NEEDEDCREATE_NEVERWhether the job is allowed to create tables.
NOCsv parsing options.
YESThe table where to put query results.
If not provided, a new table is created.
YEStrueDoes the task will failed for an empty file
NOCSVJSONAVROPARQUETORCThe source format, and possibly some parsing options, of the external data.
YESThe fully-qualified URIs that point to source data
YESWhether BigQuery should allow extra values that are not represented in the table schema.
If true, the extra values are ignored. If false, records with extra columns are treated as bad records, and if there are too many bad records, an invalid error is returned in the job result. By default unknown values are not allowed.
YESThe GCP service account to impersonate.
YESThe geographic location where the dataset should reside.
This property is experimental and might be subject to change or removed.
See Dataset Location
YESThe maximum number of bad records that BigQuery can ignore when running the job.
If the number of bad records exceeds this value, an invalid error is returned in the job result. By default, no bad record is ignored.
YESThe GCP project ID.
NOAutomatic retry for retryable BigQuery exceptions.
Some exceptions (especially rate limit) are not retried by default by BigQuery client, we use by default a transparent retry (not the kestra one) to handle this case. The default values are exponential of 5 seconds for a maximum of 15 minutes and ten attempts
YES["due to concurrent update","Retrying the job may solve the problem"]The messages which would trigger an automatic retry.
Message is tested as a substring of the full message, and is case insensitive.
YES["rateLimitExceeded","jobBackendError","internalError","jobInternalError"]The reasons which would trigger an automatic retry.
YESThe schema for the destination table.
The schema can be omitted if the destination table already exists, or if you're loading data from a Google Cloud Datastore backup (i.e. DATASTORE_BACKUP format option).
schema:
fields:
- name: colA
type: STRING
- name: colB
type: NUMERIC
See type from StandardSQLTypeName
YESALLOW_FIELD_ADDITIONALLOW_FIELD_RELAXATIONExperimental Options allowing the schema of the destination table to be updated as a side effect of the query job.
Schema update options are supported in two cases: when writeDisposition is WRITE_APPEND; when writeDisposition is WRITE_TRUNCATE and the destination table is a partition of a table, specified by partition decorators. For normal tables, WRITE_TRUNCATE will always overwrite the schema.
YES["https://www.googleapis.com/auth/cloud-platform"]The GCP scopes to be used.
YESThe GCP service account.
YESThe time partitioning field for the destination table.
YESDAYDAYHOURMONTHYEARThe time partitioning type specification for the destination table.
YESWRITE_TRUNCATEWRITE_APPENDWRITE_EMPTYThe action that should occur if the destination table already exists.
Destination table
The job id
Output rows count
NOdurationNORETRY_FAILED_TASKRETRY_FAILED_TASKCREATE_NEW_EXECUTIONNO >= 1NOdurationNOconstantNOfalseNOdurationNOdurationNORETRY_FAILED_TASKRETRY_FAILED_TASKCREATE_NEW_EXECUTIONNO >= 1NOdurationNOrandomNOfalseYESWhether BigQuery should accept rows that are missing trailing optional columns.
If true, BigQuery treats missing trailing columns as null values. If {@code false}, records with missing trailing columns are treated as bad records, and if there are too many bad records, an invalid error is returned in the job result. By default, rows with missing trailing columns are considered bad records.
YESWhether BigQuery should allow quoted data sections that contain newline characters in a CSV file.
By default quoted newline are not allowed.
YESThe character encoding of the data.
The supported values are UTF-8 or ISO-8859-1. The default value is UTF-8. BigQuery decodes the data after the raw, binary data has been split using the values set in {@link #setQuote(String)} and {@link #setFieldDelimiter(String)}.
YESThe separator for fields in a CSV file.
BigQuery converts the string to ISO-8859-1 encoding, and then uses the first byte of the encoded string to split the data in its raw, binary state. BigQuery also supports the escape sequence "\t" to specify a tab separator. The default value is a comma (',').
YESThe value that is used to quote data sections in a CSV file.
BigQuery converts the string to ISO-8859-1 encoding, and then uses the first byte of the encoded string to split the data in its raw, binary state. The default value is a double-quote ('"'). If your data does not contain quoted sections, set the property value to an empty string. If your data contains quoted newline characters, you must also set {@link #setAllowQuotedNewLines(boolean)} property to {@code true}.
YESThe number of rows at the top of a CSV file that BigQuery will skip when reading the data
The default value is 0. This property is useful if you have header rows in the file that should be skipped.
NOdurationNOdurationNORETRY_FAILED_TASKRETRY_FAILED_TASKCREATE_NEW_EXECUTIONNONO >= 1NOdurationNOexponentialNOfalseYESIf format is set to AVRO, you can interpret logical types into their corresponding types (such as TIMESTAMP) instead of only using their raw types (such as INTEGER)
The value may be null.