CreateTable CreateTable

yaml
type: "io.kestra.plugin.gcp.bigquery.CreateTable"

Create a table

Examples

yaml
id: "create_table"
type: "io.kestra.plugin.gcp.bigquery.CreateTable"
projectId: my-project
dataset: my-dataset
table: my-table
tableDefinition:
  type: TABLE
  schema:
    fields:
    - name: id
      type: INT64
    - name: name
      type: STRING
  standardTableDefinition:
    clustering:
    - id
    - name
friendlyName: new_table

Properties

dataset

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The dataset's user-defined id

table

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The table user-defined id

description

  • Type: string
  • Dynamic: ✔️
  • Required:

The user-friendly description for the table

encryptionConfiguration

The encryption configuration

expirationDuration

  • Type: string
  • Dynamic:
  • Required:
  • Format: duration

Sets the duration since now when this table expires.

If not present, the table will persist indefinitely. Expired tables will be deleted and their storage reclaimed.

friendlyName

  • Type: string
  • Dynamic: ✔️
  • Required:

The user-friendly name for the table

labels

  • Type: object
  • SubType: string
  • Dynamic: ✔️
  • Required:

Return a map for labels applied to the table

location

  • Type: string
  • Dynamic: ✔️
  • Required:

The geographic location where the dataset should reside

This property is experimental and might be subject to change or removed.

See Dataset Location

projectId

  • Type: string
  • Dynamic: ✔️
  • Required:

The GCP project id

requirePartitionFilter

  • Type: boolean
  • Dynamic:
  • Required:

Return true if a partition filter (that can be used for partition elimination) is required for queries over this table.

retryAuto

retryMessages

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required:

The message that are valid for a automatic retry.

Message is tested as a substring of the full message and case insensitive

retryReasons

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required:

The reason that are valid for a automatic retry.

scopes

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required:
  • Default: [https://www.googleapis.com/auth/cloud-platform]

The GCP scopes to used

serviceAccount

  • Type: string
  • Dynamic: ✔️
  • Required:

The GCP service account key

tableDefinition

The table definition

Outputs

creationTime

  • Type: string

The time when this table was created

datasetId

  • Type: string

The dataset's id

definition

The table definition

description

  • Type: string

The user-friendly description for the table

encryptionConfiguration

The encryption configuration

etag

  • Type: string

The hash of the table resource

expirationTime

  • Type: string

Returns the time when this table expires

If not present, the table will persist indefinitely. Expired tables will be deleted and their storage reclaimed.

friendlyName

  • Type: string

The user-friendly name for the table

generatedId

  • Type: string

The service-generated id for the table

labels

  • Type: object
  • SubType: string

Return a map for labels applied to the table

lastModifiedTime

  • Type: string

The time when this table was last modified

numBytes

  • Type: integer

The size of this table in bytes

numLongTermBytes

  • Type: integer

The number of bytes considered "long-term storage" for reduced billing purposes.

numRows

  • Type: integer

The number of rows of data in this table

projectId

  • Type: string

The project's id

requirePartitionFilter

  • Type: boolean

Return true if a partition filter (that can be used for partition elimination) is required for queries over this table.

  • Type: string

The URL that can be used to access the resource again. The returned URL can be used for get or update requests.

table

  • Type: string

The table name

Definitions

Constant

interval

  • Type: string
  • Dynamic:
  • Required: ✔️
  • Format: duration

maxAttempt

  • Type: integer
  • Dynamic:
  • Required:
  • Minimum: >= 1

maxDuration

  • Type: string
  • Dynamic:
  • Required:
  • Format: duration

warningOnRetry

  • Type: boolean
  • Dynamic:
  • Required:
  • Default: false

MaterializedViewDefinition

enableRefresh

  • Type: boolean
  • Dynamic:
  • Required:

is enable automatic refresh of the materialized view when the base table is updated

lastRefreshDate

  • Type: string
  • Dynamic:
  • Required:

Date when this materialized view was last modified

query

  • Type: string
  • Dynamic: ✔️
  • Required:

the query whose result is persisted

refreshInterval

  • Type: string
  • Dynamic:
  • Required:
  • Format: duration

the maximum frequency at which this materialized view will be refreshed

TableDefinition

externalTableDefinition

the external table definition if the type is EXTERNAL

materializedViewDefinition

the materialized view definition if the type is MATERIALIZED_VIEW

schema

  • Type: Schema
  • Dynamic:
  • Required:

the table's schema.

standardTableDefinition

the table definition if the type is TABLE

viewDefinition

the view definition if the type is VIEW

TimePartitioning

expiration

  • Type: string
  • Dynamic:
  • Required:
  • Format: duration

the number of milliseconds for which to keep the storage for a partition. When expired, the storage for the partition is reclaimed. If null, the partion does not expire.

field

  • Type: string
  • Dynamic: ✔️
  • Required:

if not set, the table is partitioned by pseudo column '_PARTITIONTIME'; if set, the table is partitioned by this field.

requirePartitionFilter

  • Type: boolean
  • Dynamic:
  • Required:

If set to true, queries over this table require a partition filter (that can be used for partition elimination) to be specified.

StandardTableDefinition

clustering

  • Type: array
  • SubType: string
  • Dynamic:
  • Required:

Returns the clustering configuration for this table. If {@code null}, the table is not clustered.

rangePartitioning

Returns the range partitioning configuration for this table. If {@code null}, the table is not range-partitioned.

streamingBuffer

Returns information on the table's streaming buffer if any exists. Returns {@code null} if no streaming buffer exists.

timePartitioning

Returns the time partitioning configuration for this table. If {@code null}, the table is not time-partitioned.

Exponential

interval

  • Type: string
  • Dynamic:
  • Required: ✔️
  • Format: duration

maxInterval

  • Type: string
  • Dynamic:
  • Required: ✔️
  • Format: duration

delayFactor

  • Type: number
  • Dynamic:
  • Required:

maxAttempt

  • Type: integer
  • Dynamic:
  • Required:
  • Minimum: >= 1

maxDuration

  • Type: string
  • Dynamic:
  • Required:
  • Format: duration

warningOnRetry

  • Type: boolean
  • Dynamic:
  • Required:
  • Default: false

Field

description

  • Type: string
  • Dynamic: ✔️
  • Required:

the field description.

mode

  • Type: string
  • Dynamic: ✔️
  • Required:
  • Possible Values:
    • NULLABLE
    • REQUIRED
    • REPEATED

the field mode.

By default Field.Mode.NULLABLE is used.

name

  • Type: string
  • Dynamic: ✔️
  • Required:

the field name.

policyTags

the policy tags for the field.

subFields

  • Type: array
  • SubType: Field
  • Dynamic: ✔️
  • Required:

the list of sub-fields if type is a LegacySQLType.RECORD. Returns null otherwise.

UserDefinedFunction

Type of UserDefinedFunction

  • Type: string
  • Dynamic: ✔️
  • Required:

If type is UserDefinedFunction.Type.INLINE this method returns a code blob. If type is UserDefinedFunction.Type.FROM_URI the method returns a Google Cloud Storage URI (e.g. gs://bucket/path)

the type of user defined function.

  • Type: string
  • Dynamic:
  • Required:
  • Possible Values:
    • INLINE
    • FROM_URI

ExternalTableDefinition

autodetect

  • Type: boolean
  • Dynamic:
  • Required:

whether automatic detection of schema and format options should be performed.

compression

  • Type: string
  • Dynamic: ✔️
  • Required:

the compression type of the data source.

formatType

  • Type: string
  • Dynamic:
  • Required:
  • Possible Values:
    • CSV
    • JSON
    • BIGTABLE
    • DATASTORE_BACKUP
    • AVRO
    • GOOGLE_SHEETS
    • PARQUET
    • ORC

the source format, and possibly some parsing options, of the external data.

ignoreUnknownValues

  • Type: boolean
  • Dynamic:
  • Required:

whether BigQuery should allow extra values that are not represented in the table schema.

If true, the extra values are ignored. If false, records with extra columns are treated as bad records, and if there are too many bad records, an invalid error is returned in the job result.

maxBadRecords

  • Type: integer
  • Dynamic:
  • Required:

the maximum number of bad records that BigQuery can ignore when reading data.

If the number of bad records exceeds this value, an invalid error is returned in the job result.

sourceUris

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required:

the fully-qualified URIs that point to your data in Google Cloud Storage.

Each URI can

  • contain one '*' wildcard character that must come after the bucket's name. Size limits related
  • to load jobs apply to external data sources, plus an additional limit of 10 GB maximum size
  • across all URIs.

StreamingBuffer

estimatedBytes

  • Type: integer
  • Dynamic:
  • Required:

estimatedRows

  • Type: integer
  • Dynamic:
  • Required:

oldestEntryTime

  • Type: integer
  • Dynamic:
  • Required:

Range

the end of range partitioning.

  • Type: integer
  • Dynamic:
  • Required:

the start of range partitioning.

  • Type: integer
  • Dynamic:
  • Required:

the width of each interval.

  • Type: integer
  • Dynamic:
  • Required:

Random

maxInterval

  • Type: string
  • Dynamic:
  • Required: ✔️
  • Format: duration

minInterval

  • Type: string
  • Dynamic:
  • Required: ✔️
  • Format: duration

maxAttempt

  • Type: integer
  • Dynamic:
  • Required:
  • Minimum: >= 1

maxDuration

  • Type: string
  • Dynamic:
  • Required:
  • Format: duration

warningOnRetry

  • Type: boolean
  • Dynamic:
  • Required:
  • Default: false

PolicyTags

the policy tags names.

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required:

RangePartitioning

the range of range partitioning.

  • Type: Range
  • Dynamic:
  • Required:

the range partitioning field.

  • Type: string
  • Dynamic: ✔️
  • Required:

Schema

the fields associated to this schema.

  • Type: array
  • SubType: Field
  • Dynamic:
  • Required:

ViewDefinition

query

  • Type: string
  • Dynamic: ✔️
  • Required:

The query whose result is persisted

viewUserDefinedFunctions

User defined functions that can be used by query. Returns null if not set.

EncryptionConfiguration

the Kms key name

  • Type: string
  • Dynamic: ✔️
  • Required: