How to use the GCP EE plugin

Run tasks on Google Cloud Batch, Cloud Run, and Compute Engine, orchestrate existing Cloud Run Jobs, and ship execution logs to Cloud Storage and Cloud Logging from Kestra flows.

Authentication

Set serviceAccount (a GCP service account JSON key) to authenticate explicitly. Set impersonatedServiceAccount (service account email) to impersonate a service account for API calls. If neither is set, application default credentials are used (GOOGLE_APPLICATION_CREDENTIALS or the environment's default service account). Optionally set scopes (default https://www.googleapis.com/auth/cloud-platform). Set projectId on each task or globally via plugin defaults. Store secrets in secrets.

Task runners

runner.Batch runs task scripts on Google Cloud Batch VMs — set projectId and region (both required). Set machineType (default e2-medium). Optionally set bucket (GCS bucket for staging inputFiles, outputFiles, and namespaceFiles), computeResource (with cpu in milliCPU, memory in MiB, and bootDisk), reservation (pre-reserved VM resource name to avoid provisioning delays), networkInterfaces (each requiring network and optionally subnetwork), maxRetryCount (0–10, default 0), and lifecyclePolicies.

Key behavioral properties: delete (default true), resume (default true), waitUntilCompletion (default PT1H), completionCheckInterval (default PT5S).

Tasks execute from the root directory — always use {{ workingDir }} or the WORKING_DIR environment variable to reference the task's working directory. Output files can also be written to {{ outputDir }} (OUTPUT_DIR) for automatic capture.

Required IAM roles: roles/batch.jobsEditor, roles/logging.viewer, roles/storage.objectAdmin on the staging bucket, and roles/iam.serviceAccountUser.

runner.CloudRun runs task scripts as Google Cloud Run jobs — set projectId and region (both required). Optionally set bucket (GCS staging bucket), resources (with cpu e.g. 1 or 1000m, and memory e.g. 512Mi, 2Gi), runtimeServiceAccount (the service account identity the Cloud Run job executes as, separate from the API authentication account), vpcAccessConnector (VPC Access Connector resource name), vpcEgress (PRIVATE_RANGES_ONLY or ALL_TRAFFIC), network and subnetwork for Direct VPC Egress, volumes (list of GCS bucket volumes with bucket, mountPath, and readOnly), and maxRetries (default 3).

Key behavioral properties: delete (default true), resume (default true), waitUntilCompletion (default PT1H).

runner.ComputeEngine runs task scripts directly on a Compute Engine VM instance — no SSH, no IAP tunnel. Set projectId and zone (both required), plus either instanceConfig (a raw instances.insert JSON object to create a new instance) or instanceName (to target an already existing instance instead). Optionally set machineType (default e2-medium, only applies when creating a new instance) and bucket (GCS bucket for staging inputFiles, outputFiles, and namespaceFiles).

The script is injected as the instance's startup-script metadata and completion is detected through a guest attribute the script writes on exit, so the target image needs bash, curl, and python3 (true of the default Debian/Ubuntu images) and enable-guest-attributes (set automatically). Targeting an existing instance via instanceName reboots it (instances.reset) to (re)trigger the script — this is destructive to anything else the instance is doing, so never point two concurrent executions at the same instanceName; ComputeEngine also never deletes or creates an instance it did not create itself, regardless of deleteInstance.

Key behavioral properties: stopInstance (default true, ignored when deleteInstance is also true or when targeting an existing instance via instanceName), deleteInstance (default true; never deletes an instanceName-targeted instance, but still controls whether that run's staged GCS blob prefix is cleaned up, regardless of instance ownership), resume (default true, reattaches to a matching Kestra-created instance instead of creating a duplicate when the Worker restarts mid-run), waitUntilCompletion (default PT1H), completionCheckInterval (default PT5S).

Required IAM roles: roles/compute.instanceAdmin.v1 (or equivalent fine-grained permissions to create, start, stop, delete instances, read guest attributes, and read the serial console) and roles/iam.serviceAccountUser if the instance runs under a dedicated service account.

Cloud Run jobs

cloudrun.Run submits a new execution of an already-existing Cloud Run Job — unlike runner.CloudRun, it never creates, updates, or deletes the Job definition, it only triggers run.jobs.run. Set projectId, region, and jobName (all required); jobName accepts either the short job name or the fully-qualified projects/{project}/locations/{region}/jobs/{job} resource path. Optionally override the Job template for this execution only with containerArgs, containerEnvironment, and taskCount (each left unset falls back to the Job template's own configuration).

Key behavioral properties: wait (default true; when false, returns immediately after submission with only executionName set) and completionCheckInterval (default PT5S). The task-level timeout property (standard on every Kestra task) bounds how long it polls for completion; defaults to one hour when unset.

Required IAM roles: roles/run.developer to submit executions and roles/logging.viewer to stream logs.

Log exporters

gcs.LogExporter ships execution logs to Google Cloud Storage — set projectId and bucket (both required). Optionally set format (default JSON, also accepts ION), maxLinesPerFile (default 100000), logFilePrefix (default kestra-log-file), and chunk (logs per request, default 1000). Requires roles/storage.objectCreator on the target bucket.

operationalsuite.LogExporter ships execution logs to Google Cloud Logging — set projectId (required). Optionally set chunk (logs per request, default 1000). Requires roles/logging.logWriter at the project level.