Google Cloud CreateTable

Google Cloud CreateTable

Certified

Create a BigQuery table

Creates a table from the provided definition, labels, encryption, and expiration settings. Fails if the table already exists.

yaml
type: io.kestra.plugin.gcp.bigquery.CreateTable
yaml
id: gcp_bq_create_table
namespace: company.team

tasks:
  - id: create_table
    type: io.kestra.plugin.gcp.bigquery.CreateTable
    projectId: my-project
    dataset: my-dataset
    table: my-table
    tableDefinition:
      type: TABLE
      schema:
        fields:
        - name: id
          type: INT64
        - name: name
          type: STRING
      standardTableDefinition:
        clustering:
        - id
        - name
    friendlyName: new_table
Properties

Dataset ID

Target dataset name

Table ID

Target table name

Table description

Encryption configuration

Optional CMEK for the table

Definitions
kmsKeyNamestring

The KMS key name.

Expiration duration

Duration from now until table expiration; if unset, table persists

Table display name

The GCP service account to impersonate

SubTypestring

Table labels

Dataset location

Optional BigQuery location for created or targeted resources. Experimental and may change; see BigQuery dataset location documentation.

Reference (ref) of the pluginDefaults to apply to this task.

The GCP project ID

Require partition filter

If true, queries must include a partition filter for partitioned tables

Automatic BigQuery retry policy

Optional custom retry policy for retryable BigQuery errors. If unset, uses an exponential backoff starting at 5s (per-attempt interval capped at 60m), with a total duration of up to 15m and a maximum of 10 attempts.

Definitions

Retry with a fixed delay between attempts.

interval*Requiredstring
Formatduration
type*Requiredobject
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse

Retry with exponentially increasing delays between attempts.

interval*Requiredstring
Formatduration
maxInterval*Requiredstring
Formatduration
type*Requiredobject
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
delayFactornumber
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse

Retry with a random delay within a configurable range between attempts.

maxInterval*Requiredstring
Formatduration
minInterval*Requiredstring
Formatduration
type*Requiredobject
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse
SubTypestring
Default["due to concurrent update","Retrying the job may solve the problem","Retrying may solve the problem"]

Retry message substrings

Case-insensitive substrings that, if found in the error message, trigger an automatic retry

SubTypestring
Default["rateLimitExceeded","jobBackendError","backendError","internalError","jobInternalError"]

Retry reasons

BigQuery error reasons that trigger an automatic retry; evaluated against error reason strings

SubTypestring
Default["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used

The GCP service account

Table definition

The table definition (standard, external, view, etc.). Required when creating a table; optional when updating, where the current definition is preserved unless overridden.

Definitions
externalTableDefinition

The external table definition if the type is EXTERNAL

autodetectbooleanstring

Whether automatic detection of schema and format options should be performed

compressionstring

The compression type of the data source

formatTypestring
Possible Values
CSVJSONBIGTABLEDATASTORE_BACKUPAVROGOOGLE_SHEETSPARQUETORC

The source format, and possibly some parsing options, of the external data

ignoreUnknownValuesbooleanstring

Whether BigQuery should allow extra values that are not represented in the table schema

If true, the extra values are ignored. If false, records with extra columns are treated as bad records, and if there are too many bad records, an invalid error is returned in the job result.

maxBadRecordsintegerstring

The maximum number of bad records that BigQuery can ignore when reading data

If the number of bad records exceeds this value, an invalid error is returned in the job result.

sourceUrisarray
SubTypestring

The fully-qualified URIs that point to your data in Google Cloud Storage

Each URI can contain one '*' wildcard character that must come after the bucket's name. Size limits related to load jobs apply to external data sources, plus an additional limit of 10 GB maximum size across all URIs.

materializedViewDefinition

The materialized view definition if the type is MATERIALIZED_VIEW

enableRefreshbooleanstring

Whether automatic refresh is enabled for the materialized view when the base table is updated

lastRefreshDatestring
Formatdate-time

Date when this materialized view was last refreshed

querystring

The query whose result is persisted

refreshIntervalstring

The maximum frequency at which this materialized view will be refreshed

schema

The table's schema

fieldsarray

The fields associated with this schema.

descriptionstring

The field description

modestring
Possible Values
NULLABLEREQUIREDREPEATED

The field mode

By default, Field.Mode.NULLABLE is used.

namestring

The field name

policyTags

The policy tags for the field

subFieldsarray

The list of sub-fields if type is a record/struct type (StandardSQLTypeName.STRUCT). Null otherwise.

typestring
Possible Values
BOOLINT64FLOAT64NUMERICBIGNUMERICSTRINGBYTESSTRUCTARRAYTIMESTAMPDATETIMEDATETIMEGEOGRAPHYJSONINTERVALRANGE

The field type

standardTableDefinition

The table definition if the type is TABLE

clusteringarray
SubTypestring

The clustering configuration for this table. If null, the table is not clustered

rangePartitioning

The range partitioning configuration for this table. If null, the table is not range-partitioned

fieldstring

The range partitioning field.

range

The range of range partitioning.

streamingBuffer

Information on the table's streaming buffer, if it exists. Null if no streaming buffer exists

estimatedBytesinteger
estimatedRowsinteger
oldestEntryTimeinteger
timePartitioning

The time partitioning configuration for this table. If null, the table is not time-partitioned

expirationstring

The duration for which to keep the storage for a partition. When expired, the storage for the partition is reclaimed. If null, the partition does not expire

fieldstring

If not set, the table is partitioned by pseudo column '_PARTITIONTIME'; if set, the table is partitioned by this field

requirePartitionFilterbooleanstring

If set to true, queries over this table require a partition filter (that can be used for partition elimination) to be specified

typestring
Possible Values
DAYHOURMONTHYEAR

The time partitioning type

typestring
Possible Values
TABLEVIEWMATERIALIZED_VIEWEXTERNALMODEL

The table's type

viewDefinition

The view definition if the type is VIEW

querystring

The query whose result is persisted

viewUserDefinedFunctionsarray

User defined functions that can be used by query. Null if not set

contentstring

The content of the user defined function

If type is INLINE, this is a code blob. If type is FROM_URI, this is a Google Cloud Storage URI (e.g. gs://bucket/path).

typestring
Possible Values
INLINEFROM_URI

The type of user defined function.

Formatdate-time

Creation time

Dataset ID

Table definition

Definitions
externalTableDefinition
autodetectboolean
compressionstring
formatTypestring
Possible Values
CSVJSONBIGTABLEDATASTORE_BACKUPAVROGOOGLE_SHEETSPARQUETORC
ignoreUnknownValuesboolean
maxBadRecordsinteger
sourceUrisarray
SubTypestring
materializedViewDefinition
enableRefreshboolean
lastRefreshDatestring
Formatdate-time
querystring
refreshIntervalstring
Formatduration
schema
fieldsarray
descriptionstring
modestring
Possible Values
NULLABLEREQUIREDREPEATED
namestring
policyTags
subFieldsarray
typestring
Possible Values
BOOLINT64FLOAT64NUMERICBIGNUMERICSTRINGBYTESSTRUCTARRAYTIMESTAMPDATETIMEDATETIMEGEOGRAPHYJSONINTERVALRANGE
standardTableDefinition
clusteringarray
SubTypestring
rangePartitioning
fieldstring
range
streamingBuffer
estimatedBytesinteger
estimatedRowsinteger
oldestEntryTimeinteger
timePartitioning
expirationstring
Formatduration
fieldstring
requirePartitionFilterboolean
typestring
Possible Values
DAYHOURMONTHYEAR
typestring
Possible Values
TABLEVIEWMATERIALIZED_VIEWEXTERNALMODEL
viewDefinition
querystring
viewUserDefinedFunctionsarray
contentstring
typestring
Possible Values
INLINEFROM_URI

Description

Encryption configuration

Definitions
kmsKeyNamestring

Table etag

Formatdate-time

Expiration time

If absent, the table persists indefinitely; expired tables are deleted

Friendly name

Service-generated table id

SubTypestring

Labels

Formatdate-time

Last modified time

Size in bytes

Long-term storage bytes

Bytes billed at long-term storage rates

Row count

Project ID

Require partition filter

True when queries must specify a partition filter for partition elimination

Table name

More Plugins in this Category

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks

PostgreSQL
PostgreSQL

Tasks that connect to PostgreSQL via JDBC to run queries and loads.

Data

6 tasks

25 blueprints