Copy
type: "io.kestra.plugin.gcp.bigquery.Copy"
Copy a BigQuery table or partition to other one.
Examples
id: gcp_bq_copy
namespace: company.team
tasks:
- id: copy
type: io.kestra.plugin.gcp.bigquery.Copy
operationType: COPY
sourceTables:
- "my_project.my_dataset.my_table$20130908"
destinationTable: "my_project.my_dataset.my_table"
Properties
destinationTable
- Type: string
- Dynamic: ✔️
- Required: ✔️
The destination table.
If not provided a new table is created.
operationType
- Type: string
- Dynamic: ✔️
- Required: ✔️
- Possible Values:
COPY
SNAPSHOT
RESTORE
CLONE
Sets the supported operation types in table copy job.
COPY
: The source and destination table have the same table type.
SNAPSHOT
: The source table type is TABLE and the destination table type is SNAPSHOT.RESTORE
: The source table type is SNAPSHOT and the destination table type is TABLE.CLONE
: The source and destination table have the same table type, but only bill for unique data.
sourceTables
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ✔️
The source tables.
Can be table or partitions.
createDisposition
- Type: string
- Dynamic: ❌
- Required: ❌
- Possible Values:
CREATE_IF_NEEDED
CREATE_NEVER
Whether the job is allowed to create tables.
dryRun
- Type: boolean
- Dynamic: ❌
- Required: ❌
- Default:
false
Whether the job has to be dry run or not.
A valid query will mostly return an empty response with some processing statistics, while an invalid query will return the same error as it would if it were an actual run.
impersonatedServiceAccount
- Type: string
- Dynamic: ✔️
- Required: ❌
The GCP service account to impersonate.
jobTimeout
- Type: string
- Dynamic: ❌
- Required: ❌
- Format:
duration
Job timeout.
If this time limit is exceeded, BigQuery may attempt to terminate the job.
labels
- Type: object
- SubType: string
- Dynamic: ✔️
- Required: ❌
The labels associated with this job.
You can use these to organize and group your jobs. Label keys and values can be no longer than 63 characters, can only contain lowercase letters, numeric characters, underscores and dashes. International characters are allowed. Label values are optional. Label keys must start with a letter and each label in the list must have a different key.
location
- Type: string
- Dynamic: ✔️
- Required: ❌
The geographic location where the dataset should reside.
This property is experimental and might be subject to change or removed.
See Dataset Location
projectId
- Type: string
- Dynamic: ✔️
- Required: ❌
The GCP project ID.
retryAuto
- Type:
- Dynamic: ❌
- Required: ❌
retryMessages
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
- Default:
[due to concurrent update, Retrying the job may solve the problem]
The messages which would trigger an automatic retry.
Message is tested as a substring of the full message, and is case insensitive.
retryReasons
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
- Default:
[rateLimitExceeded, jobBackendError, internalError, jobInternalError]
The reasons which would trigger an automatic retry.
scopes
- Type: array
- SubType: string
- Dynamic: ✔️
- Required: ❌
- Default:
[https://www.googleapis.com/auth/cloud-platform]
The GCP scopes to be used.
serviceAccount
- Type: string
- Dynamic: ✔️
- Required: ❌
The GCP service account.
writeDisposition
- Type: string
- Dynamic: ❌
- Required: ❌
- Possible Values:
WRITE_TRUNCATE
WRITE_APPEND
WRITE_EMPTY
The action that should occur if the destination table already exists.
Outputs
jobId
- Type: string
- Required: ❌
The job id
Definitions
io.kestra.core.models.tasks.retrys.Constant
Properties
interval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
type
- Type: string
- Dynamic: ❓
- Required: ✔️
- Default:
constant
behavior
- Type: string
- Dynamic: ❓
- Required: ❌
- Default:
RETRY_FAILED_TASK
- Possible Values:
RETRY_FAILED_TASK
CREATE_NEW_EXECUTION
maxAttempt
- Type: integer
- Dynamic: ❓
- Required: ❌
- Minimum:
>= 1
maxDuration
- Type: string
- Dynamic: ❓
- Required: ❌
- Format:
duration
warningOnRetry
- Type: boolean
- Dynamic: ❓
- Required: ❌
- Default:
false
io.kestra.core.models.tasks.retrys.Random
Properties
maxInterval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
minInterval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
type
- Type: string
- Dynamic: ❓
- Required: ✔️
- Default:
random
behavior
- Type: string
- Dynamic: ❓
- Required: ❌
- Default:
RETRY_FAILED_TASK
- Possible Values:
RETRY_FAILED_TASK
CREATE_NEW_EXECUTION
maxAttempt
- Type: integer
- Dynamic: ❓
- Required: ❌
- Minimum:
>= 1
maxDuration
- Type: string
- Dynamic: ❓
- Required: ❌
- Format:
duration
warningOnRetry
- Type: boolean
- Dynamic: ❓
- Required: ❌
- Default:
false
io.kestra.core.models.tasks.retrys.Exponential
Properties
interval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
maxInterval
- Type: string
- Dynamic: ❓
- Required: ✔️
- Format:
duration
type
- Type: string
- Dynamic: ❓
- Required: ✔️
- Default:
exponential
behavior
- Type: string
- Dynamic: ❓
- Required: ❌
- Default:
RETRY_FAILED_TASK
- Possible Values:
RETRY_FAILED_TASK
CREATE_NEW_EXECUTION
delayFactor
- Type: number
- Dynamic: ❓
- Required: ❌
maxAttempt
- Type: integer
- Dynamic: ❓
- Required: ❌
- Minimum:
>= 1
maxDuration
- Type: string
- Dynamic: ❓
- Required: ❌
- Format:
duration
warningOnRetry
- Type: boolean
- Dynamic: ❓
- Required: ❌
- Default:
false
Metrics
num.child.jobs
- Type: counter
duration
- Type: timer
Was this page helpful?