Google Cloud LoadFromGcs

Google Cloud LoadFromGcs

Certified

Load GCS objects into BigQuery

Runs a BigQuery load job from one or more GCS URIs into the destination table. Supports wildcard paths, format-specific options, and standard load limits. Table must exist unless schema is supplied with a write disposition that creates it.

yaml
type: io.kestra.plugin.gcp.bigquery.LoadFromGcs

Load an avro file from a gcs bucket

yaml
id: gcp_bq_load_from_gcs
namespace: company.team

tasks:
  - id: http_download
    type: io.kestra.plugin.core.http.Download
    uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv

  - id: csv_to_ion
    type: io.kestra.plugin.serdes.csv.CsvToIon
    from: "{{ outputs.http_download.uri }}"
    header: true

  - id: ion_to_avro
    type: io.kestra.plugin.serdes.avro.IonToAvro
    from: "{{ outputs.csv_to_ion.uri }}"
    schema: |
      {
        "type": "record",
        "name": "Order",
        "namespace": "com.example.order",
        "fields": [
          {"name": "order_id", "type": "int"},
          {"name": "customer_name", "type": "string"},
          {"name": "customer_email", "type": "string"},
          {"name": "product_id", "type": "int"},
          {"name": "price", "type": "double"},
          {"name": "quantity", "type": "int"},
          {"name": "total", "type": "double"}
        ]
      }

  - id: upload_to_gcs
    type: io.kestra.plugin.gcp.gcs.Upload
    from: "{{ outputs.ion_to_avro.uri }}"
    to: "gs://my-bucket/orders.avro"

  - id: load_from_gcs
    type: io.kestra.plugin.gcp.bigquery.LoadFromGcs
    from:
      - "{{ outputs.upload_to_gcs.uri }}"
    destinationTable: "my-project.my_dataset.my_table"
    format: AVRO
    avroOptions:
      useAvroLogicalTypes: true

Load a csv file with a defined schema

yaml
id: gcp_bq_load_files_test
namespace: company.team

tasks:
  - id: load_files_test
    type: io.kestra.plugin.gcp.bigquery.LoadFromGcs
    destinationTable: "myDataset.myTable"
    ignoreUnknownValues: true
    schema:
      fields:
        - name: colA
          type: STRING
        - name: colB
          type: NUMERIC
        - name: colC
          type: STRING
    format: CSV
    csvOptions:
      allowJaggedRows: true
      encoding: UTF-8
      fieldDelimiter: ","
    from:
      - gs://myBucket/myFile.csv
Properties

Autodetect source options

Experimental. Lets BigQuery infer schema/options for CSV or JSON sources.

Avro parsing options

Definitions
useAvroLogicalTypesbooleanstring

If format is set to AVRO, you can interpret logical types into their corresponding types (such as TIMESTAMP) instead of only using their raw types (such as INTEGER)

The value may be null.

SubTypestring

Clustering fields

Possible Values
CREATE_IF_NEEDEDCREATE_NEVER

Create disposition

Whether the job may create the destination table

CSV parsing options

Definitions
allowJaggedRowsbooleanstring

Whether BigQuery should accept rows that are missing trailing optional columns

If true, BigQuery treats missing trailing columns as null values. If false, records with missing trailing columns are treated as bad records, and if there are too many bad records, an invalid error is returned in the job result. By default, rows with missing trailing columns are considered bad records.

allowQuotedNewLinesbooleanstring

Whether BigQuery should allow quoted data sections that contain newline characters in a CSV file

By default quoted newline are not allowed.

encodingstring

The character encoding of the data

The supported values are UTF-8 or ISO-8859-1. The default value is UTF-8. BigQuery decodes the data after the raw, binary data has been split using the values set in the quote and fieldDelimiter properties.

fieldDelimiterstring

The separator for fields in a CSV file

BigQuery converts the string to ISO-8859-1 encoding, and then uses the first byte of the encoded string to split the data in its raw, binary state. BigQuery also supports the escape sequence "\t" to specify a tab separator. The default value is a comma (',').

quotestring

The value that is used to quote data sections in a CSV file

BigQuery converts the string to ISO-8859-1 encoding, and then uses the first byte of the encoded string to split the data in its raw, binary state. The default value is a double-quote ('"'). If your data does not contain quoted sections, set the property value to an empty string. If your data contains quoted newline characters, you must also set the allowQuotedNewLines property to true.

skipLeadingRowsintegerstring

The number of rows at the top of a CSV file that BigQuery will skip when reading the data

The default value is 0. This property is useful if you have header rows in the file that should be skipped.

Destination table

Target table for the load job; creation depends on createDisposition

Possible Values
CSVJSONAVROPARQUETORC

Source format

SubTypestring

Google Cloud Storage source data

The fully-qualified URIs that point to source data in Google Cloud Storage (e.g. gs://bucket/path). Each URI can contain one '*' wildcard character and it must come after the 'bucket' name.

Ignore unknown values

If true, extra columns are skipped; if false, extra columns count as bad records. Default is false.

The GCP service account to impersonate

Dataset location

Optional BigQuery location for created or targeted resources. Experimental and may change; see BigQuery dataset location documentation.

Max bad records

Number of bad records allowed before the job fails; default 0

Reference (ref) of the pluginDefaults to apply to this task.

The GCP project ID

Automatic BigQuery retry policy

Optional custom retry policy for retryable BigQuery errors. If unset, uses an exponential backoff starting at 5s (per-attempt interval capped at 60m), with a total duration of up to 15m and a maximum of 10 attempts.

Definitions

Retry with a fixed delay between attempts.

interval*Requiredstring
Formatduration
type*Requiredobject
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse

Retry with exponentially increasing delays between attempts.

interval*Requiredstring
Formatduration
maxInterval*Requiredstring
Formatduration
type*Requiredobject
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
delayFactornumber
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse

Retry with a random delay within a configurable range between attempts.

maxInterval*Requiredstring
Formatduration
minInterval*Requiredstring
Formatduration
type*Requiredobject
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse
SubTypestring
Default["due to concurrent update","Retrying the job may solve the problem","Retrying may solve the problem"]

Retry message substrings

Case-insensitive substrings that, if found in the error message, trigger an automatic retry

SubTypestring
Default["rateLimitExceeded","jobBackendError","backendError","internalError","jobInternalError"]

Retry reasons

BigQuery error reasons that trigger an automatic retry; evaluated against error reason strings

Destination schema

Table schema definition; may be omitted when loading into an existing table or supported backup formats

SubTypestring
Possible Values
ALLOW_FIELD_ADDITIONALLOW_FIELD_RELAXATION

Schema update options

Experimental. Applies only with WRITE_APPEND or partitioned WRITE_TRUNCATE destinations.

SubTypestring
Default["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used

The GCP service account

Time partitioning field

DefaultDAY
Possible Values
DAYHOURMONTHYEAR

Time partitioning type

Defaults to DAY when partitioning is configured

Possible Values
WRITE_TRUNCATEWRITE_TRUNCATE_DATAWRITE_APPENDWRITE_EMPTY

Write disposition

Action if destination exists (e.g., WRITE_APPEND, WRITE_TRUNCATE)

Destination table

The job id

Output rows count

Unitrecords

the number of bad records reported in a job.

The time it took for the task to run.

Unitbytes

The number of bytes of source data in a load job.

Unitfiles

The number of source files in a load job.

Unitbytes

The size of the data loaded by a load job so far, in bytes.

Unitrecords

The number of rows loaded by a load job so far.

More Plugins in this Category

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks

OPA (Open Policy Agent)
OPA (Open Policy Agent)
Enterprise

Tasks to manage Open Policy Agent policies and decisions, including upload, evaluation, and partial evaluation.

Infrastructure

5 tasks

3 blueprints