Google Cloud CopyPartitions
CertifiedCopy BigQuery partitions to another table
Google Cloud CopyPartitions
Copy BigQuery partitions to another table
Lists partitions in a source table over the given date range and copies them to a destination table using a BigQuery copy job. Supports create/write dispositions and dry run.
type: io.kestra.plugin.gcp.bigquery.CopyPartitionsExamples
id: gcp_bq_copy_partitions
namespace: company.team
tasks:
- id: copy_partitions
type: io.kestra.plugin.gcp.bigquery.CopyPartitions
projectId: my-project
dataset: my-dataset
table: my-table
destinationTable: my-dest-table
partitionType: DAY
from: "{{ now() | dateAdd(-30, 'DAYS') }}"
to: "{{ now() | dateAdd(-7, 'DAYS') }}"
Properties
dataset *Requiredstring
Dataset ID
Target dataset name
from *Requiredstring
Partition start
Inclusive start value. Use an integer for RANGE partitions or an ISO datetime for date/time partitions.
partitionType *Requiredstring
DAYHOURMONTHYEARRANGEPartition type
DAY, HOUR, MONTH, YEAR, or RANGE
table *Requiredstring
Table ID
Target table name
to *Requiredstring
Partition end
Inclusive end value. Use an integer for RANGE partitions or an ISO datetime for date/time partitions.
createDisposition string
CREATE_IF_NEEDEDCREATE_NEVERCreate disposition
BigQuery create disposition applied to the copy job
destinationTable string
Destination table
Target table receiving the copied partitions
dryRun booleanstring
falseDry run
If true, validates the job without executing
impersonatedServiceAccount string
The GCP service account to impersonate
jobTimeout string
Job timeout
Optional maximum duration for the copy job
labels object
Job labels
location string
Dataset location
Optional BigQuery location for created or targeted resources. Experimental and may change; see BigQuery dataset location documentation.
pluginDefaultsRef Non-dynamicstring
Reference (ref) of the pluginDefaults to apply to this task.
projectId string
The GCP project ID
retryAuto Non-dynamic
Automatic BigQuery retry policy
Optional custom retry policy for retryable BigQuery errors. If unset, uses an exponential backoff starting at 5s (per-attempt interval capped at 60m), with a total duration of up to 15m and a maximum of 10 attempts.
Constant retry
Retry with a fixed delay between attempts.
durationRETRY_FAILED_TASKRETRY_FAILED_TASKCREATE_NEW_EXECUTION>= 1durationfalseExponential retry
Retry with exponentially increasing delays between attempts.
durationdurationRETRY_FAILED_TASKRETRY_FAILED_TASKCREATE_NEW_EXECUTION>= 1durationfalseRandom retry
Retry with a random delay within a configurable range between attempts.
durationdurationRETRY_FAILED_TASKRETRY_FAILED_TASKCREATE_NEW_EXECUTION>= 1durationfalseretryMessages array
["due to concurrent update","Retrying the job may solve the problem","Retrying may solve the problem"]Retry message substrings
Case-insensitive substrings that, if found in the error message, trigger an automatic retry
retryReasons array
["rateLimitExceeded","jobBackendError","backendError","internalError","jobInternalError"]Retry reasons
BigQuery error reasons that trigger an automatic retry; evaluated against error reason strings
scopes array
["https://www.googleapis.com/auth/cloud-platform"]The GCP scopes to be used
serviceAccount string
The GCP service account
skipEmpty booleanstring
falseSkip empty partitions
If true, the task completes successfully without running a copy job when no partitions are found in the specified range. If false (the default), the task will fail when there are no partitions to copy.
writeDisposition string
WRITE_TRUNCATEWRITE_TRUNCATE_DATAWRITE_APPENDWRITE_EMPTYWrite disposition
BigQuery write disposition applied to the copy job
Outputs
datasetId string
Dataset ID
jobId string
Job ID
partitions array
Partitions copied
projectId string
Project ID
table string
Table name
Metrics
size counter
The number of partitions copied.
