Google Cloud ExtractToGcs

Google Cloud ExtractToGcs

Certified

Export BigQuery table to GCS

Runs an extract job from a table or partition to one or more GCS URIs. Supports CSV/JSON/AVRO, optional compression, custom delimiter, and AVRO logical types. Prints a header row by default.

yaml
type: io.kestra.plugin.gcp.bigquery.ExtractToGcs

Extract a BigQuery table to a GCS bucket

yaml
id: gcp_bq_extract_to_gcs
namespace: company.team

tasks:
  - id: extract_to_gcs
    type: io.kestra.plugin.gcp.bigquery.ExtractToGcs
    destinationUris:
      - "gs://bucket_name/filename.csv"
    sourceTable: "my-project.my_dataset.my_table"
    format: CSV
    fieldDelimiter: ';'
    printHeader: true
Properties

Compression

Optional compression; if unset, files are uncompressed

SubTypestring

Destination URIs

One or more gs:// URIs to receive the export; wildcards allowed

Field delimiter

Delimiter between fields; defaults to comma

Export format

CSV by default; AVRO and other formats supported

The GCP service account to impersonate

Job timeout (ms)

Optional timeout; BigQuery may terminate the job if exceeded

Job labels

Dataset location

Optional BigQuery location for created or targeted resources. Experimental and may change; see BigQuery dataset location documentation.

Reference (ref) of the pluginDefaults to apply to this task.

Print header

If true (default), include a header row in the export

The GCP project ID

Automatic BigQuery retry policy

Optional custom retry policy for retryable BigQuery errors. If unset, uses an exponential backoff starting at 5s (per-attempt interval capped at 60m), with a total duration of up to 15m and a maximum of 10 attempts.

Definitions

Retry with a fixed delay between attempts.

interval*Requiredstring
Formatduration
type*Requiredobject
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse

Retry with exponentially increasing delays between attempts.

interval*Requiredstring
Formatduration
maxInterval*Requiredstring
Formatduration
type*Requiredobject
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
delayFactornumber
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse

Retry with a random delay within a configurable range between attempts.

maxInterval*Requiredstring
Formatduration
minInterval*Requiredstring
Formatduration
type*Requiredobject
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse
SubTypestring
Default["due to concurrent update","Retrying the job may solve the problem","Retrying may solve the problem"]

Retry message substrings

Case-insensitive substrings that, if found in the error message, trigger an automatic retry

SubTypestring
Default["rateLimitExceeded","jobBackendError","backendError","internalError","jobInternalError"]

Retry reasons

BigQuery error reasons that trigger an automatic retry; evaluated against error reason strings

SubTypestring
Default["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used

The GCP service account

Source table

Fully-qualified table (optionally partition) to export

Use AVRO logical types

For AVRO exports, map supported columns to AVRO logical types

SubTypestring

Destination URIs

SubTypeinteger

Number of extracted files

Job ID

Source table

The time it took for the job to run.

The number of files extracted to GCS.

More Plugins in this Category

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks

PostgreSQL
PostgreSQL

Tasks that connect to PostgreSQL via JDBC to run queries and loads.

Data

6 tasks

25 blueprints