pluginsplugin-gcptasksbigqueryio.kestra.plugin.gcp.bigquery.createtable

CreateTable

yaml

type: "io.kestra.plugin.gcp.bigquery.CreateTable"

Create a table

Examples

yaml

id: "create_table"
type: "io.kestra.plugin.gcp.bigquery.CreateTable"
projectId: my-project
dataset: my-dataset
table: my-table
tableDefinition:
  type: TABLE
  schema:
    fields:
    - name: id
      type: INT64
    - name: name
      type: STRING
  standardTableDefinition:
    clustering:
    - id
    - name
friendlyName: new_table

Properties

`dataset`

Type: string
Dynamic: ✔️
Required: ✔️

The dataset's user-defined id

`table`

Type: string
Dynamic: ✔️
Required: ✔️

The table user-defined id

`description`

Type: string
Dynamic: ✔️
Required: ❌

The user-friendly description for the table

`encryptionConfiguration`

Type: EncryptionConfiguration
Dynamic: ❓
Required: ❌

The encryption configuration

`expirationDuration`

Type: string
Dynamic: ❓
Required: ❌
Format: duration

Sets the duration since now when this table expires.

If not present, the table will persist indefinitely. Expired tables will be deleted and their storage reclaimed.

`friendlyName`

Type: string
Dynamic: ✔️
Required: ❌

The user-friendly name for the table

`labels`

Type: object
SubType: string
Dynamic: ✔️
Required: ❌

Return a map for labels applied to the table

`location`

Type: string
Dynamic: ✔️
Required: ❌

The geographic location where the dataset should reside

This property is experimental and might be subject to change or removed.

See Dataset Location

`projectId`

Type: string
Dynamic: ✔️
Required: ❌

The GCP project id

`requirePartitionFilter`

Type: boolean
Dynamic: ❓
Required: ❌

Return true if a partition filter (that can be used for partition elimination) is required for queries over this table.

`retryAuto`

Type:Constant Exponential Random
Dynamic: ❓
Required: ❌

`retryMessages`

Type: array
SubType: string
Dynamic: ✔️
Required: ❌
Default: [due to concurrent update, Retrying the job may solve the problem]

The message that are valid for a automatic retry.

Message is tested as a substring of the full message and case insensitive

`retryReasons`

Type: array
SubType: string
Dynamic: ✔️
Required: ❌
Default: [rateLimitExceeded, jobBackendError, internalError, jobInternalError]

The reason that are valid for a automatic retry.

`scopes`

Type: array
SubType: string
Dynamic: ✔️
Required: ❌
Default: [https://www.googleapis.com/auth/cloud-platform]

The GCP scopes to used

`serviceAccount`

Type: string
Dynamic: ✔️
Required: ❌

The GCP service account key

`tableDefinition`

Type: TableDefinition
Dynamic: ❓
Required: ❌

The table definition

Outputs

`creationTime`

Type: string

The time when this table was created

`datasetId`

Type: string

The dataset's id

`definition`

Type: TableDefinition

The table definition

`description`

Type: string

The user-friendly description for the table

`encryptionConfiguration`

Type: EncryptionConfiguration

The encryption configuration

`etag`

Type: string

The hash of the table resource

`expirationTime`

Type: string

Returns the time when this table expires

If not present, the table will persist indefinitely. Expired tables will be deleted and their storage reclaimed.

`friendlyName`

Type: string

The user-friendly name for the table

`generatedId`

Type: string

The service-generated id for the table

`labels`

Type: object
SubType: string

Return a map for labels applied to the table

`lastModifiedTime`

Type: string

The time when this table was last modified

`numBytes`

Type: integer

The size of this table in bytes

`numLongTermBytes`

Type: integer

The number of bytes considered "long-term storage" for reduced billing purposes.

`numRows`

Type: integer

The number of rows of data in this table

`projectId`

Type: string

The project's id

`requirePartitionFilter`

Type: boolean

Return true if a partition filter (that can be used for partition elimination) is required for queries over this table.

`selfLink`

Type: string

The URL that can be used to access the resource again. The returned URL can be used for get or update requests.

`table`

Type: string

The table name

Definitions

`Constant`

`interval`

Type: string
Dynamic: ❓
Required: ✔️
Format: duration

`type`

Type: string
Dynamic: ❓
Required: ✔️
Default: constant

`maxAttempt`

Type: integer
Dynamic: ❓
Required: ❌
Minimum: >= 1

`maxDuration`

Type: string
Dynamic: ❓
Required: ❌
Format: duration

`warningOnRetry`

Type: boolean
Dynamic: ❓
Required: ❌
Default: false

`MaterializedViewDefinition`

`enableRefresh`

Type: boolean
Dynamic: ❌
Required: ❌

is enable automatic refresh of the materialized view when the base table is updated

`lastRefreshDate`

Type: string
Dynamic: ❓
Required: ❌

Date when this materialized view was last modified

`query`

Type: string
Dynamic: ✔️
Required: ❌

the query whose result is persisted

`refreshInterval`

Type: string
Dynamic: ❌
Required: ❌
Format: duration

the maximum frequency at which this materialized view will be refreshed

`TableDefinition`

`externalTableDefinition`

Type: ExternalTableDefinition
Dynamic: ❓
Required: ❌

the external table definition if the type is EXTERNAL

`materializedViewDefinition`

Type: MaterializedViewDefinition
Dynamic: ❓
Required: ❌

the materialized view definition if the type is MATERIALIZED_VIEW

`schema`

Type: Schema
Dynamic: ❓
Required: ❌

the table's schema.

`standardTableDefinition`

Type: StandardTableDefinition
Dynamic: ❓
Required: ❌

the table definition if the type is TABLE

`type`

Type: string
Dynamic: ❓
Required: ❌
Possible Values:
- TABLE
- VIEW
- MATERIALIZED_VIEW
- EXTERNAL
- MODEL

the table's type.

`viewDefinition`

Type: ViewDefinition
Dynamic: ❓
Required: ❌

the view definition if the type is VIEW

`TimePartitioning`

`expiration`

Type: string
Dynamic: ❌
Required: ❌
Format: duration

the number of milliseconds for which to keep the storage for a partition. When expired, the storage for the partition is reclaimed. If null, the partion does not expire.

`field`

Type: string
Dynamic: ✔️
Required: ❌

if not set, the table is partitioned by pseudo column '_PARTITIONTIME'; if set, the table is partitioned by this field.

`requirePartitionFilter`

Type: boolean
Dynamic: ❌
Required: ❌

If set to true, queries over this table require a partition filter (that can be used for partition elimination) to be specified.

`type`

Type: string
Dynamic: ❌
Required: ❌
Possible Values:
- DAY
- HOUR
- MONTH
- YEAR

the time partitioning type.

`StandardTableDefinition`

`clustering`

Type: array
SubType: string
Dynamic: ❓
Required: ❌

Returns the clustering configuration for this table. If {@code null}, the table is not clustered.

`rangePartitioning`

Type: RangePartitioning
Dynamic: ❓
Required: ❌

Returns the range partitioning configuration for this table. If {@code null}, the table is not range-partitioned.

`streamingBuffer`

Type: StreamingBuffer
Dynamic: ❓
Required: ❌

Returns information on the table's streaming buffer if any exists. Returns {@code null} if no streaming buffer exists.

`timePartitioning`

Type: TimePartitioning
Dynamic: ❓
Required: ❌

Returns the time partitioning configuration for this table. If {@code null}, the table is not time-partitioned.

`Exponential`

`interval`

Type: string
Dynamic: ❓
Required: ✔️
Format: duration

`maxInterval`

Type: string
Dynamic: ❓
Required: ✔️
Format: duration

`type`

Type: string
Dynamic: ❓
Required: ✔️
Default: exponential

`delayFactor`

Type: number
Dynamic: ❓
Required: ❌

`maxAttempt`

Type: integer
Dynamic: ❓
Required: ❌
Minimum: >= 1

`maxDuration`

Type: string
Dynamic: ❓
Required: ❌
Format: duration

`warningOnRetry`

Type: boolean
Dynamic: ❓
Required: ❌
Default: false

`Field`

`description`

Type: string
Dynamic: ✔️
Required: ❌

the field description.

`mode`

Type: string
Dynamic: ✔️
Required: ❌
Possible Values:
- NULLABLE
- REQUIRED
- REPEATED

the field mode.

By default Field.Mode.NULLABLE is used.

`name`

Type: string
Dynamic: ✔️
Required: ❌

the field name.

`policyTags`

Type: PolicyTags
Dynamic: ✔️
Required: ❌

the policy tags for the field.

`subFields`

Type: array
SubType: Field
Dynamic: ✔️
Required: ❌

the list of sub-fields if type is a LegacySQLType.RECORD. Returns null otherwise.

`type`

Type: string
Dynamic: ❌
Required: ❌
Possible Values:
- BOOL
- INT64
- FLOAT64
- NUMERIC
- BIGNUMERIC
- STRING
- BYTES
- STRUCT
- ARRAY
- TIMESTAMP
- DATE
- TIME
- DATETIME
- GEOGRAPHY
- JSON
- INTERVAL

the field name.

`UserDefinedFunction`

`Type of UserDefinedFunction`

Type: string
Dynamic: ✔️
Required: ❌

If type is UserDefinedFunction.Type.INLINE this method returns a code blob. If type is UserDefinedFunction.Type.FROM_URI the method returns a Google Cloud Storage URI (e.g. gs://bucket/path)

`the type of user defined function.`

Type: string
Dynamic: ❌
Required: ❌
Possible Values:
- INLINE
- FROM_URI

`ExternalTableDefinition`

`autodetect`

Type: boolean
Dynamic: ❓
Required: ❌

whether automatic detection of schema and format options should be performed.

`compression`

Type: string
Dynamic: ✔️
Required: ❌

the compression type of the data source.

`formatType`

Type: string
Dynamic: ❓
Required: ❌
Possible Values:
- CSV
- JSON
- BIGTABLE
- DATASTORE_BACKUP
- AVRO
- GOOGLE_SHEETS
- PARQUET
- ORC

the source format, and possibly some parsing options, of the external data.

`ignoreUnknownValues`

Type: boolean
Dynamic: ❓
Required: ❌

whether BigQuery should allow extra values that are not represented in the table schema.

If true, the extra values are ignored. If false, records with extra columns are treated as bad records, and if there are too many bad records, an invalid error is returned in the job result.

`maxBadRecords`

Type: integer
Dynamic: ❓
Required: ❌

the maximum number of bad records that BigQuery can ignore when reading data.

If the number of bad records exceeds this value, an invalid error is returned in the job result.

`sourceUris`

Type: array
SubType: string
Dynamic: ✔️
Required: ❌

the fully-qualified URIs that point to your data in Google Cloud Storage.

Each URI can

contain one '*' wildcard character that must come after the bucket's name. Size limits related
to load jobs apply to external data sources, plus an additional limit of 10 GB maximum size
across all URIs.