🚀 New! Kestra raises $3 million to grow Learn more

pluginsplugin-gcptasksbigqueryio.kestra.plugin.gcp.bigquery.copypartitions

​Copy​Partitions

CopyPartitions CopyPartitions

yaml
type: "io.kestra.plugin.gcp.bigquery.CopyPartitions"

Copy partitions between interval to another table

Examples

yaml
id: "copy_partitions"
type: "io.kestra.plugin.gcp.bigquery.CopyPartitions"
projectId: my-project
dataset: my-dataset
table: my-table
destinationTable: my-dest-table
partitionType: DAY
from: "{{ now() | dateAdd(-30, 'DAYS') }}"
to: "{{ now() | dateAdd(-7, 'DAYS') }}"

Properties

dataset

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The dataset's user-defined id

from

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The inclusive starting date or integer

If the partition :- is a numeric range, must be a valid integer

  • is a date, must a valid datetime like null

partitionType

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️
  • Possible Values:
    • DAY
    • HOUR
    • MONTH
    • YEAR
    • RANGE

The partition type of the table

table

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The table user-defined id

to

  • Type: string
  • Dynamic: ✔️
  • Required: ✔️

The inclusive ending date or integer

If the partition :- is a numeric range, must be a valid integer

  • is a date, must a valid datetime like null

createDisposition

  • Type: string
  • Dynamic: ❌
  • Required: ❌
  • Possible Values:
    • CREATE_IF_NEEDED
    • CREATE_NEVER

Whether the job is allowed to create tables

destinationTable

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The table where to put query results

If not provided a new table is created.

dryRun

  • Type: boolean
  • Dynamic: ❌
  • Required: ❌
  • Default: false

Sets whether the job has to be dry run or no.

A valid query will return a mostly empty response with some processing statistics, while an invalid query will return the same error it would if it wasn't a dry run.

jobTimeout

  • Type: string
  • Dynamic: ❌
  • Required: ❌
  • Format: duration

Job timeout.

If this time limit is exceeded, BigQuery may attempt to terminate the job.

labels

  • Type: object
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌

The labels associated with this job.

You can use these to organize and group your jobs. Label keys and values can be no longer than 63 characters, can only contain lowercase letters, numeric characters, underscores and dashes. International characters are allowed. Label values are optional. Label keys must start with a letter and each label in the list must have a different key.

location

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The geographic location where the dataset should reside

This property is experimental and might be subject to change or removed.

See Dataset Location

projectId

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The GCP project id

retryAuto

retryMessages

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: [due to concurrent update, Retrying the job may solve the problem]

The message that are valid for a automatic retry.

Message is tested as a substring of the full message and case insensitive

retryReasons

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: [rateLimitExceeded, jobBackendError, internalError, jobInternalError]

The reason that are valid for a automatic retry.

scopes

  • Type: array
  • SubType: string
  • Dynamic: ✔️
  • Required: ❌
  • Default: [https://www.googleapis.com/auth/cloud-platform]

The GCP scopes to used

serviceAccount

  • Type: string
  • Dynamic: ✔️
  • Required: ❌

The GCP service account key

writeDisposition

  • Type: string
  • Dynamic: ❌
  • Required: ❌
  • Possible Values:
    • WRITE_TRUNCATE
    • WRITE_APPEND
    • WRITE_EMPTY

The action that should occur if the destination table already exists

Outputs

datasetId

  • Type: string

The dataset's id

jobId

  • Type: string

The job id

partitions

  • Type: array
  • SubType: string

Partitions copied

projectId

  • Type: string

The project's id

table

  • Type: string

The table name

Definitions

Constant

interval

  • Type: string
  • Dynamic: ❓
  • Required: ✔️
  • Format: duration

type

  • Type: string
  • Dynamic: ❓
  • Required: ✔️
  • Default: constant

maxAttempt

  • Type: integer
  • Dynamic: ❓
  • Required: ❌
  • Minimum: >= 1

maxDuration

  • Type: string
  • Dynamic: ❓
  • Required: ❌
  • Format: duration

warningOnRetry

  • Type: boolean
  • Dynamic: ❓
  • Required: ❌
  • Default: false

Random

maxInterval

  • Type: string
  • Dynamic: ❓
  • Required: ✔️
  • Format: duration

minInterval

  • Type: string
  • Dynamic: ❓
  • Required: ✔️
  • Format: duration

type

  • Type: string
  • Dynamic: ❓
  • Required: ✔️
  • Default: random

maxAttempt

  • Type: integer
  • Dynamic: ❓
  • Required: ❌
  • Minimum: >= 1

maxDuration

  • Type: string
  • Dynamic: ❓
  • Required: ❌
  • Format: duration

warningOnRetry

  • Type: boolean
  • Dynamic: ❓
  • Required: ❌
  • Default: false

Exponential

interval

  • Type: string
  • Dynamic: ❓
  • Required: ✔️
  • Format: duration

maxInterval

  • Type: string
  • Dynamic: ❓
  • Required: ✔️
  • Format: duration

type

  • Type: string
  • Dynamic: ❓
  • Required: ✔️
  • Default: exponential

delayFactor

  • Type: number
  • Dynamic: ❓
  • Required: ❌

maxAttempt

  • Type: integer
  • Dynamic: ❓
  • Required: ❌
  • Minimum: >= 1

maxDuration

  • Type: string
  • Dynamic: ❓
  • Required: ❌
  • Format: duration

warningOnRetry

  • Type: boolean
  • Dynamic: ❓
  • Required: ❌
  • Default: false