PluginsGoogle CloudTasksBigqueryCopyPartitions

CopyPartitions

CopyPartitions

yaml

type: "io.kestra.plugin.gcp.bigquery.CopyPartitions"

Copy partitions between interval to another table

Examples

yaml

id: "copy_partitions"
type: "io.kestra.plugin.gcp.bigquery.CopyPartitions"
projectId: my-project
dataset: my-dataset
table: my-table
destinationTable: my-dest-table
partitionType: DAY
from: "{{ now() | dateAdd(-30, 'DAYS') }}"
to: "{{ now() | dateAdd(-7, 'DAYS') }}"

Properties

`dataset`

Type: string
Dynamic: ✔️
Required: ✔️

The dataset's user-defined ID.

`from`

Type: string
Dynamic: ✔️
Required: ✔️

The inclusive starting date or integer.

If the partition :

is a numeric range, must be a valid integer
is a date, must a valid datetime like

`partitionType`

Type: string
Dynamic: ✔️
Required: ✔️
Possible Values:
- DAY
- HOUR
- MONTH
- YEAR
- RANGE

The partition type of the table

`table`

Type: string
Dynamic: ✔️
Required: ✔️

The table's user-defined ID.

`to`

Type: string
Dynamic: ✔️
Required: ✔️

The inclusive ending date or integer.

If the partition :

is a numeric range, must be a valid integer
is a date, must a valid datetime like

`createDisposition`

Type: string
Dynamic: ❌
Required: ❌
Possible Values:
- CREATE_IF_NEEDED
- CREATE_NEVER

Whether the job is allowed to create tables.

`destinationTable`

Type: string
Dynamic: ✔️
Required: ❌

The table where to put query results.

If not provided, a new table is created.

`dryRun`

Type: boolean
Dynamic: ❌
Required: ❌
Default: false

Whether the job has to be dry run or not.

A valid query will mostly return an empty response with some processing statistics, while an invalid query will return the same error as it would if it were an actual run.

`jobTimeout`

Type: string
Dynamic: ❌
Required: ❌
Format: duration

Job timeout.

If this time limit is exceeded, BigQuery may attempt to terminate the job.

`labels`

Type: object
SubType: string
Dynamic: ✔️
Required: ❌

The labels associated with this job.

You can use these to organize and group your jobs. Label keys and values can be no longer than 63 characters, can only contain lowercase letters, numeric characters, underscores and dashes. International characters are allowed. Label values are optional. Label keys must start with a letter and each label in the list must have a different key.

`location`

Type: string
Dynamic: ✔️
Required: ❌

The geographic location where the dataset should reside.

This property is experimental and might be subject to change or removed.

See Dataset Location

`projectId`

Type: string
Dynamic: ✔️
Required: ❌

The GCP project ID.

`retryAuto`

Type:
Dynamic: ❓
Required: ❌

`retryMessages`

Type: array
SubType: string
Dynamic: ✔️
Required: ❌
Default: [due to concurrent update, Retrying the job may solve the problem]

The messages which would trigger an automatic retry.

Message is tested as a substring of the full message, and is case insensitive.

`retryReasons`

Type: array
SubType: string
Dynamic: ✔️
Required: ❌
Default: [rateLimitExceeded, jobBackendError, internalError, jobInternalError]

The reasons which would trigger an automatic retry.

`scopes`

Type: array
SubType: string
Dynamic: ✔️
Required: ❌
Default: [https://www.googleapis.com/auth/cloud-platform]

The GCP scopes to be used.

`serviceAccount`

Type: string
Dynamic: ✔️
Required: ❌

The GCP service account key.

`writeDisposition`

Type: string
Dynamic: ❌
Required: ❌
Possible Values:
- WRITE_TRUNCATE
- WRITE_APPEND
- WRITE_EMPTY

The action that should occur if the destination table already exists.

Outputs

`datasetId`

Type: string
Dynamic: ❓
Required: ❌

The dataset's id

`jobId`

Type: string
Dynamic: ❓
Required: ❌

The job id

`partitions`

Type: array
SubType: string
Dynamic: ❓
Required: ❌

Partitions copied

`projectId`

Type: string
Dynamic: ❓
Required: ❌

The project's id

`table`

Type: string
Dynamic: ❓
Required: ❌

The table name

Definitions

`io.kestra.core.models.tasks.retrys.Constant`

Properties

`interval`

Type: string
Dynamic: ❓
Required: ✔️
Format: duration

`type`

Type: string
Dynamic: ❓
Required: ✔️
Default: constant

`behavior`

Type: string
Dynamic: ❓
Required: ❌
Default: RETRY_FAILED_TASK
Possible Values:
- RETRY_FAILED_TASK
- CREATE_NEW_EXECUTION

`maxAttempt`

Type: integer
Dynamic: ❓
Required: ❌
Minimum: >= 1

`maxDuration`

Type: string
Dynamic: ❓
Required: ❌
Format: duration

`warningOnRetry`

Type: boolean
Dynamic: ❓
Required: ❌
Default: false

`io.kestra.core.models.tasks.retrys.Random`

Properties

`maxInterval`

Type: string
Dynamic: ❓
Required: ✔️
Format: duration

`minInterval`

Type: string
Dynamic: ❓
Required: ✔️
Format: duration

`type`

Type: string
Dynamic: ❓
Required: ✔️
Default: random

`behavior`

Type: string
Dynamic: ❓
Required: ❌
Default: RETRY_FAILED_TASK
Possible Values:
- RETRY_FAILED_TASK
- CREATE_NEW_EXECUTION

`maxAttempt`

Type: integer
Dynamic: ❓
Required: ❌
Minimum: >= 1

`maxDuration`

Type: string
Dynamic: ❓
Required: ❌
Format: duration

`warningOnRetry`

Type: boolean
Dynamic: ❓
Required: ❌
Default: false

`io.kestra.core.models.tasks.retrys.Exponential`

Properties

`interval`

Type: string
Dynamic: ❓
Required: ✔️
Format: duration

`maxInterval`

Type: string
Dynamic: ❓
Required: ✔️
Format: duration

`type`

Type: string
Dynamic: ❓
Required: ✔️
Default: exponential

`behavior`

Type: string
Dynamic: ❓
Required: ❌
Default: RETRY_FAILED_TASK
Possible Values:
- RETRY_FAILED_TASK
- CREATE_NEW_EXECUTION

`delayFactor`

Type: number
Dynamic: ❓
Required: ❌

`maxAttempt`

Type: integer
Dynamic: ❓
Required: ❌
Minimum: >= 1

`maxDuration`

Type: string
Dynamic: ❓
Required: ❌
Format: duration

`warningOnRetry`

Type: boolean
Dynamic: ❓
Required: ❌
Default: false

Metrics

`size`

Type: counter