CreateTable
type: "io.kestra.plugin.gcp.bigquery.CreateTable"
Create a table
Examples
id: "create_table"
type: "io.kestra.plugin.gcp.bigquery.CreateTable"
projectId: my-project
dataset: my-dataset
table: my-table
tableDefinition:
type: TABLE
schema:
fields:
- name: id
type: INT64
- name: name
type: STRING
standardTableDefinition:
clustering:
- id
- name
friendlyName: new_table
Properties
dataset
- Type: string
- Dynamic: ✔️
- Required: ✔️
The dataset's user-defined id
table
- Type: string
- Dynamic: ✔️
- Required: ✔️
The table user-defined id
description
- Type: string
- Dynamic: ✔️
- Required: ❌
The user-friendly description for the table
encryptionConfiguration
- Type: EncryptionConfiguration
- Dynamic: ❓
- Required: ❌
The encryption configuration
expirationDuration
- Type: string
- Dynamic: ❓
- Required: ❌
- Format:
duration
Sets the duration since now when this table expires.
If not present, the table will persist indefinitely. Expired tables will be deleted and their storage reclaimed.
friendlyName
- Type: string
- Dynamic: ✔️
- Required: ❌
The user-friendly name for the table
labels
- Type: object
- SubType: string
- Dynamic: ✔️
- Required: ❌
Return a map for labels applied to the table
location
- Type: string
- Dynamic: ✔️
- Required: ❌
The geographic location where the dataset should reside
This property is experimental and might be subject to change or removed.
See Dataset Location
projectId
- Type: string
- Dynamic: ✔️
- Required: ❌
The GCP project id
requirePartitionFilter
- Type: boolean
- Dynamic: ❓
- Required: ❌
Return true if a partition filter (that can be used for partition elimination) is required for queries over this table.
retryAuto
- Type:ConstantExponentialRandom
- Dynamic: ❓
- Required: ❌
retryMessages
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
The message that are valid for a automatic retry.
Message is tested as a substring of the full message and case insensitive
retryReasons
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
The reason that are valid for a automatic retry.
scopes
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
- Default:
[https://www.googleapis.com/auth/cloud-platform]
The GCP scopes to used
serviceAccount
- Type: string
- Dynamic: ✔️
- Required: ❌
The GCP service account key
tableDefinition
- Type: TableDefinition
- Dynamic: ❓
- Required: ❌
The table definition
Outputs
creationTime
- Type: string
The time when this table was created
datasetId
- Type: string
The dataset's id
definition
- Type: TableDefinition
The table definition
description
- Type: string
The user-friendly description for the table
encryptionConfiguration
- Type: EncryptionConfiguration
The encryption configuration
etag
- Type: string
The hash of the table resource
expirationTime
- Type: string
Returns the time when this table expires
If not present, the table will persist indefinitely. Expired tables will be deleted and their storage reclaimed.
friendlyName
- Type: string
The user-friendly name for the table
generatedId
- Type: string
The service-generated id for the table
labels
- Type: object
- SubType: string
Return a map for labels applied to the table
lastModifiedTime
- Type: string
The time when this table was last modified
numBytes
- Type: integer
The size of this table in bytes
numLongTermBytes
- Type: integer
The number of bytes considered "long-term storage" for reduced billing purposes.
numRows
- Type: integer
The number of rows of data in this table
projectId
- Type: string
The project's id
requirePartitionFilter
- Type: boolean
Return true if a partition filter (that can be used for partition elimination) is required for queries over this table.
selfLink
- Type: string
The URL that can be used to access the resource again. The returned URL can be used for get or update requests.
table
- Type: string
The table name
Definitions
Constant
interval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
maxAttempt
- Type: integer
- Dynamic: ❓
- Required: ❌
- Minimum:
>= 1
maxDuration
- Type: string
- Dynamic: ❓
- Required: ❌
- Format:
duration
warningOnRetry
- Type: boolean
- Dynamic: ❓
- Required: ❌
- Default:
false
MaterializedViewDefinition
enableRefresh
- Type: boolean
- Dynamic: ❌
- Required: ❌
is enable automatic refresh of the materialized view when the base table is updated
lastRefreshDate
- Type: string
- Dynamic: ❓
- Required: ❌
Date when this materialized view was last modified
query
- Type: string
- Dynamic: ✔️
- Required: ❌
the query whose result is persisted
refreshInterval
- Type: string
- Dynamic: ❌
- Required: ❌
- Format:
duration
the maximum frequency at which this materialized view will be refreshed
TableDefinition
externalTableDefinition
- Type: ExternalTableDefinition
- Dynamic: ❓
- Required: ❌
the external table definition if the type is
EXTERNAL
materializedViewDefinition
- Type: MaterializedViewDefinition
- Dynamic: ❓
- Required: ❌
the materialized view definition if the type is
MATERIALIZED_VIEW
schema
- Type: Schema
- Dynamic: ❓
- Required: ❌
the table's schema.
standardTableDefinition
- Type: StandardTableDefinition
- Dynamic: ❓
- Required: ❌
the table definition if the type is
TABLE
viewDefinition
- Type: ViewDefinition
- Dynamic: ❓
- Required: ❌
the view definition if the type is
VIEW
TimePartitioning
expiration
- Type: string
- Dynamic: ❌
- Required: ❌
- Format:
duration
the number of milliseconds for which to keep the storage for a partition. When expired, the storage for the partition is reclaimed. If null, the partion does not expire.
field
- Type: string
- Dynamic: ✔️
- Required: ❌
if not set, the table is partitioned by pseudo column '_PARTITIONTIME'; if set, the table is partitioned by this field.
requirePartitionFilter
- Type: boolean
- Dynamic: ❌
- Required: ❌
If set to true, queries over this table require a partition filter (that can be used for partition elimination) to be specified.
StandardTableDefinition
clustering
- Type: array
- SubType: string
- Dynamic: ❓
- Required: ❌
Returns the clustering configuration for this table. If {@code null}, the table is not clustered.
rangePartitioning
- Type: RangePartitioning
- Dynamic: ❓
- Required: ❌
Returns the range partitioning configuration for this table. If {@code null}, the table is not range-partitioned.
streamingBuffer
- Type: StreamingBuffer
- Dynamic: ❓
- Required: ❌
Returns information on the table's streaming buffer if any exists. Returns {@code null} if no streaming buffer exists.
timePartitioning
- Type: TimePartitioning
- Dynamic: ❓
- Required: ❌
Returns the time partitioning configuration for this table. If {@code null}, the table is not time-partitioned.
Exponential
interval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
maxInterval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
delayFactor
- Type: number
- Dynamic: ❓
- Required: ❌
maxAttempt
- Type: integer
- Dynamic: ❓
- Required: ❌
- Minimum:
>= 1
maxDuration
- Type: string
- Dynamic: ❓
- Required: ❌
- Format:
duration
warningOnRetry
- Type: boolean
- Dynamic: ❓
- Required: ❌
- Default:
false
Field
description
- Type: string
- Dynamic: ✔️
- Required: ❌
the field description.
mode
- Type: string
- Dynamic: ✔️
- Required: ❌
- Possible Values:
NULLABLE
REQUIRED
REPEATED
the field mode.
By default Field.Mode.NULLABLE
is used.
name
- Type: string
- Dynamic: ✔️
- Required: ❌
the field name.
policyTags
- Type: PolicyTags
- Dynamic: ✔️
- Required: ❌
the policy tags for the field.
subFields
- Type: array
- SubType: Field
- Dynamic: ✔️
- Required: ❌
the list of sub-fields if
type
is aLegacySQLType.RECORD
. Returns null otherwise.
UserDefinedFunction
Type of UserDefinedFunction
- Type: string
- Dynamic: ✔️
- Required: ❌
If type
is UserDefinedFunction.Type.INLINE this method returns a code blob.
If type
is UserDefinedFunction.Type.FROM_URI the method returns a Google Cloud Storage URI (e.g. gs://bucket/path)
the type of user defined function.
- Type: string
- Dynamic: ❌
- Required: ❌
- Possible Values:
INLINE
FROM_URI
ExternalTableDefinition
autodetect
- Type: boolean
- Dynamic: ❓
- Required: ❌
whether automatic detection of schema and format options should be performed.
compression
- Type: string
- Dynamic: ✔️
- Required: ❌
the compression type of the data source.
formatType
- Type: string
- Dynamic: ❓
- Required: ❌
- Possible Values:
CSV
JSON
BIGTABLE
DATASTORE_BACKUP
AVRO
GOOGLE_SHEETS
PARQUET
ORC
the source format, and possibly some parsing options, of the external data.
ignoreUnknownValues
- Type: boolean
- Dynamic: ❓
- Required: ❌
whether BigQuery should allow extra values that are not represented in the table schema.
If true, the extra values are ignored. If false, records with extra columns are treated as bad records, and if there are too many bad records, an invalid error is returned in the job result.
maxBadRecords
- Type: integer
- Dynamic: ❓
- Required: ❌
the maximum number of bad records that BigQuery can ignore when reading data.
If the number of bad records exceeds this value, an invalid error is returned in the job result.
sourceUris
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
the fully-qualified URIs that point to your data in Google Cloud Storage.
Each URI can
- contain one '*' wildcard character that must come after the bucket's name. Size limits related
- to load jobs apply to external data sources, plus an additional limit of 10 GB maximum size
- across all URIs.
StreamingBuffer
estimatedBytes
- Type: integer
- Dynamic: ❓
- Required: ❌
estimatedRows
- Type: integer
- Dynamic: ❓
- Required: ❌
oldestEntryTime
- Type: integer
- Dynamic: ❓
- Required: ❌
Range
the end of range partitioning.
- Type: integer
- Dynamic: ❌
- Required: ❌
the start of range partitioning.
- Type: integer
- Dynamic: ❌
- Required: ❌
the width of each interval.
- Type: integer
- Dynamic: ❌
- Required: ❌
Random
maxInterval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
minInterval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
maxAttempt
- Type: integer
- Dynamic: ❓
- Required: ❌
- Minimum:
>= 1
maxDuration
- Type: string
- Dynamic: ❓
- Required: ❌
- Format:
duration
warningOnRetry
- Type: boolean
- Dynamic: ❓
- Required: ❌
- Default:
false
PolicyTags
the policy tags names.
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
RangePartitioning
the range of range partitioning.
- Type: Range
- Dynamic: ❓
- Required: ❌
the range partitioning field.
- Type: string
- Dynamic: ✔️
- Required: ❌
Schema
the fields associated to this schema.
- Type: array
- SubType: Field
- Dynamic: ❓
- Required: ❌
ViewDefinition
query
- Type: string
- Dynamic: ✔️
- Required: ❌
The query whose result is persisted
viewUserDefinedFunctions
- Type: array
- SubType: UserDefinedFunction
- Dynamic: ❓
- Required: ❌
User defined functions that can be used by query. Returns null if not set.
EncryptionConfiguration
the Kms key name
- Type: string
- Dynamic: ✔️
- Required: ❌