Google Cloud CreateDataset

Google Cloud CreateDataset

Certified

Create or update a BigQuery dataset

Creates a dataset with optional location, labels, ACLs, and defaults. If it exists, behavior follows ifExists (default ERROR; UPDATE replaces metadata, SKIP returns existing).

yaml
type: io.kestra.plugin.gcp.bigquery.CreateDataset

Create a dataset if not exists

yaml
id: gcp_bq_create_dataset
namespace: company.team

tasks:
  - id: create_dataset
    type: io.kestra.plugin.gcp.bigquery.CreateDataset
    name: "my_dataset"
    location: "EU"
    ifExists: "SKIP"
Properties

Dataset ID

Required dataset identifier

Access controls

List of dataset ACL entries (see AccessControl)

Definitions
entity*Required

The GCP entity

type*Requiredstring
Possible Values
DOMAINGROUPUSERIAM_MEMBER

The type of the entity (USER, GROUP, DOMAIN or IAM_MEMBER)

value*Requiredstring

The value for the entity

For example, user email if the type is USER.

role*Requiredstring
Possible Values
READERWRITEROWNER

The role to assign to the entity

Default table encryption key

CMEK applied to newly created tables unless overridden

Definitions
kmsKeyNamestring

Default partition expiration (ms)

Applied to new partitioned tables only; overrides defaultTableLifetime for partitions

Default table expiration (ms)

Minimum 3,600,000 ms (1h). Applies to new tables only; existing tables keep their own expiration. Experimental.

Dataset description

User-friendly description; supports templating

Dataset display name

DefaultERROR
Possible Values
ERRORUPDATESKIP

Existing dataset policy

ERROR by default. Use UPDATE to apply new metadata or SKIP to leave the dataset unchanged.

The GCP service account to impersonate

Dataset labels

Dataset location

Optional BigQuery location; experimental and may change. See BigQuery dataset location docs.

Reference (ref) of the pluginDefaults to apply to this task.

The GCP project ID

Automatic BigQuery retry policy

Optional custom retry policy for retryable BigQuery errors. If unset, uses an exponential backoff starting at 5s (per-attempt interval capped at 60m), with a total duration of up to 15m and a maximum of 10 attempts.

Definitions

Retry with a fixed delay between attempts.

interval*Requiredstring
Formatduration
type*Requiredobject
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse

Retry with exponentially increasing delays between attempts.

interval*Requiredstring
Formatduration
maxInterval*Requiredstring
Formatduration
type*Requiredobject
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
delayFactornumber
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse

Retry with a random delay within a configurable range between attempts.

maxInterval*Requiredstring
Formatduration
minInterval*Requiredstring
Formatduration
type*Requiredobject
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse
SubTypestring
Default["due to concurrent update","Retrying the job may solve the problem","Retrying may solve the problem"]

Retry message substrings

Case-insensitive substrings that, if found in the error message, trigger an automatic retry

SubTypestring
Default["rateLimitExceeded","jobBackendError","backendError","internalError","jobInternalError"]

Retry reasons

BigQuery error reasons that trigger an automatic retry; evaluated against error reason strings

SubTypestring
Default["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used

The GCP service account

Dataset ID

Dataset description

Dataset display name

Dataset location

Optional BigQuery location; experimental and may change

GCP project ID

More Plugins in this Category

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks

PostgreSQL
PostgreSQL

Tasks that connect to PostgreSQL via JDBC to run queries and loads.

Data

6 tasks

25 blueprints