Google Cloud BigQuery CopyPartitions

Google Cloud BigQuery CopyPartitions

Certified

Copy BigQuery partitions to another table

Lists partitions in a source table over the given date range and copies them to a destination table using a BigQuery copy job. Supports create/write dispositions and dry run.

yaml
type: io.kestra.plugin.gcp.bigquery.CopyPartitions
yaml
id: gcp_bq_copy_partitions
namespace: company.team

tasks:
  - id: copy_partitions
    type: io.kestra.plugin.gcp.bigquery.CopyPartitions
    projectId: my-project
    dataset: my-dataset
    table: my-table
    destinationTable: my-dest-table
    partitionType: DAY
    from: "{{ now() | dateAdd(-30, 'DAYS') }}"
    to: "{{ now() | dateAdd(-7, 'DAYS') }}"
Installation

Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command:

kestractl plugins get io.kestra.plugin:plugin-gcp --compatible-for $KESTRA_VERSION

For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.

Properties

Dataset ID

Target dataset name

Partition start

Inclusive start value. Use an integer for RANGE partitions or an ISO datetime for date/time partitions.

Possible Values
DAYHOURMONTHYEARRANGE

Partition type

DAY, HOUR, MONTH, YEAR, or RANGE

Table ID

Target table name

Partition end

Inclusive end value. Use an integer for RANGE partitions or an ISO datetime for date/time partitions.

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions
assetFailureBehaviorstring
Possible Values
IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

id*string
Min length1
typestring
outputs

The assets produced as outputs.

id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*string
Min length1

Custom asset type

descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
Possible Values
CREATE_IF_NEEDEDCREATE_NEVER

Create disposition

BigQuery create disposition applied to the copy job

Destination table

Target table receiving the copied partitions

Defaultfalse

Dry run

If true, validates the job without executing

The GCP service account to impersonate

Job timeout

Optional maximum duration for the copy job

Job labels

Dataset location

Optional BigQuery location for created or targeted resources. Experimental and may change; see BigQuery dataset location documentation.

The GCP project ID

Automatic BigQuery retry policy

Optional custom retry policy for retryable BigQuery errors. If unset, uses an exponential backoff starting at 5s (per-attempt interval capped at 60m) with a maximum of 10 attempts and no overall time limit, so long-running queries are still retried. Set maxDuration or task timeout to cap the total time.

Definitions

Retry with a fixed delay between attempts.

interval*string
Formatduration
type*object
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse

Retry with exponentially increasing delays between attempts.

interval*string
Formatduration
maxInterval*string
Formatduration
type*object
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
delayFactornumber
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse

Retry with a random delay within a configurable range between attempts.

maxInterval*string
Formatduration
minInterval*string
Formatduration
type*object
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse
SubTypestring
Default["due to concurrent update","Retrying the job may solve the problem","Retrying may solve the problem","[CONNECTION_ERROR]","This stub requires a minimum remaining timeout"]

Retry message substrings

Case-insensitive substrings that, if found in the error message, trigger an automatic retry.

SubTypestring
Default["rateLimitExceeded","jobBackendError","backendError","internalError","jobInternalError"]

Retry reasons

BigQuery error reasons (matched case-insensitively) that trigger an automatic retry.

SubTypestring
Default["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used

The GCP service account

When unset or rendered to an empty value, Application Default Credentials are used (e.g. GOOGLE_APPLICATION_CREDENTIALS or the runtime's attached identity).

Defaultfalse

Skip empty partitions

If true, the task completes successfully without running a copy job when no partitions are found in the specified range. If false (the default), the task will fail when there are no partitions to copy.

Audience of the Workload Identity Federation provider

The full resource name of the workload identity pool provider, //iam.googleapis.com/projects/<number>/locations/global/workloadIdentityPools/<pool>/providers/<provider>. Required with workloadIdentityToken, and must match the aud claim of the token.

OIDC token for Workload Identity Federation

When set together with workloadIdentityAudience, the token is exchanged at Google STS for federated credentials and no service account key is needed: Google verifies it against the identity provider that issued it. Use {{ oidcToken(audience='…') }} to have Kestra Enterprise mint one for the running execution. Combine with impersonatedServiceAccount to act as a service account the pool may impersonate, and set projectId explicitly since no key file carries it. Cannot be combined with serviceAccount.

Possible Values
WRITE_TRUNCATEWRITE_TRUNCATE_DATAWRITE_APPENDWRITE_EMPTY

Write disposition

BigQuery write disposition applied to the copy job

Dataset ID

Job ID

SubTypestring

Partitions copied

Project ID

Table name

The number of partitions copied.