CopyPartitions
type: "io.kestra.plugin.gcp.bigquery.CopyPartitions"
Copy partitions between interval to another table
Examples
id: "copy_partitions"
type: "io.kestra.plugin.gcp.bigquery.CopyPartitions"
projectId: my-project
dataset: my-dataset
table: my-table
destinationTable: my-dest-table
partitionType: DAY
from: "{{ now() | dateAdd(-30, 'DAYS') }}"
to: "{{ now() | dateAdd(-7, 'DAYS') }}"
Properties
dataset
- Type: string
- Dynamic: ✔️
- Required: ✔️
The dataset's user-defined id
from
- Type: string
- Dynamic: ✔️
- Required: ✔️
The inclusive starting date or integer
If the partition :- is a numeric range, must be a valid integer
- is a date, must a valid datetime like
null
partitionType
- Type: string
- Dynamic: ✔️
- Required: ✔️
- Possible Values:
DAY
HOUR
MONTH
YEAR
RANGE
The partition type of the table
table
- Type: string
- Dynamic: ✔️
- Required: ✔️
The table user-defined id
to
- Type: string
- Dynamic: ✔️
- Required: ✔️
The inclusive ending date or integer
If the partition :- is a numeric range, must be a valid integer
- is a date, must a valid datetime like
null
createDisposition
- Type: string
- Dynamic: ❌
- Required: ❌
- Possible Values:
CREATE_IF_NEEDED
CREATE_NEVER
Whether the job is allowed to create tables
destinationTable
- Type: string
- Dynamic: ✔️
- Required: ❌
The table where to put query results
If not provided a new table is created.
dryRun
- Type: boolean
- Dynamic: ❌
- Required: ❌
- Default:
false
Sets whether the job has to be dry run or no.
A valid query will return a mostly empty response with some processing statistics, while an invalid query will return the same error it would if it wasn't a dry run.
jobTimeout
- Type: string
- Dynamic: ❌
- Required: ❌
- Format:
duration
Job timeout.
If this time limit is exceeded, BigQuery may attempt to terminate the job.
labels
- Type: object
- SubType: string
- Dynamic: ✔️
- Required: ❌
The labels associated with this job.
You can use these to organize and group your jobs. Label keys and values can be no longer than 63 characters, can only contain lowercase letters, numeric characters, underscores and dashes. International characters are allowed. Label values are optional. Label keys must start with a letter and each label in the list must have a different key.
location
- Type: string
- Dynamic: ✔️
- Required: ❌
The geographic location where the dataset should reside
This property is experimental and might be subject to change or removed.
See Dataset Location
projectId
- Type: string
- Dynamic: ✔️
- Required: ❌
The GCP project id
retryAuto
- Type:ConstantExponentialRandom
- Dynamic: ❓
- Required: ❌
retryMessages
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
- Default:
[due to concurrent update, Retrying the job may solve the problem]
The message that are valid for a automatic retry.
Message is tested as a substring of the full message and case insensitive
retryReasons
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
- Default:
[rateLimitExceeded, jobBackendError, internalError, jobInternalError]
The reason that are valid for a automatic retry.
scopes
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
- Default:
[https://www.googleapis.com/auth/cloud-platform]
The GCP scopes to used
serviceAccount
- Type: string
- Dynamic: ✔️
- Required: ❌
The GCP service account key
writeDisposition
- Type: string
- Dynamic: ❌
- Required: ❌
- Possible Values:
WRITE_TRUNCATE
WRITE_APPEND
WRITE_EMPTY
The action that should occur if the destination table already exists
Outputs
datasetId
- Type: string
The dataset's id
jobId
- Type: string
The job id
partitions
- Type: array
- SubType: string
Partitions copied
projectId
- Type: string
The project's id
table
- Type: string
The table name
Definitions
Constant
interval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
type
- Type: string
- Dynamic: ❓
- Required: ✔️
- Default:
constant
maxAttempt
- Type: integer
- Dynamic: ❓
- Required: ❌
- Minimum:
>= 1
maxDuration
- Type: string
- Dynamic: ❓
- Required: ❌
- Format:
duration
warningOnRetry
- Type: boolean
- Dynamic: ❓
- Required: ❌
- Default:
false
Random
maxInterval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
minInterval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
type
- Type: string
- Dynamic: ❓
- Required: ✔️
- Default:
random
maxAttempt
- Type: integer
- Dynamic: ❓
- Required: ❌
- Minimum:
>= 1
maxDuration
- Type: string
- Dynamic: ❓
- Required: ❌
- Format:
duration
warningOnRetry
- Type: boolean
- Dynamic: ❓
- Required: ❌
- Default:
false
Exponential
interval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
maxInterval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
type
- Type: string
- Dynamic: ❓
- Required: ✔️
- Default:
exponential
delayFactor
- Type: number
- Dynamic: ❓
- Required: ❌
maxAttempt
- Type: integer
- Dynamic: ❓
- Required: ❌
- Minimum:
>= 1
maxDuration
- Type: string
- Dynamic: ❓
- Required: ❌
- Format:
duration
warningOnRetry
- Type: boolean
- Dynamic: ❓
- Required: ❌
- Default:
false