Google Cloud ComputeEngine

Google Cloud ComputeEngine

Certified
Enterprise Edition

Run tasks on a Google Compute Engine VM

Enterprise-only task runner that runs task scripts directly on a Compute Engine VM instance — no SSH, no IAP tunnel. The script is injected as the instance's startup-script metadata, its stdout/stderr is mirrored by the guest environment to the serial console (port 1) and streamed back from there, and completion is detected through a guest attribute (kestra/status) the script writes when it exits. This requires enable-guest-attributes (set automatically by this runner) and an instance image that provides bash, curl, and python3 — true of the default Debian/Ubuntu Compute Engine images.

You need the roles/compute.instanceAdmin.v1 role (or equivalent fine-grained permissions) to create, start, stop, delete instances, read guest attributes, and read the serial console, plus roles/iam.serviceAccountUser if the instance runs under a dedicated service account.

By default a new instance is created from instanceConfig — a raw JSON object matching the instances.insert request body (at minimum disks and networkInterfaces) — then stopped and/or deleted after the run according to stopInstance/deleteInstance. Set instanceName instead to target an already existing instance: since a metadata startup-script only runs on boot, ComputeEngine reboots (instances.reset) that instance to (re)trigger the script if it is already running — this interrupts anything else the instance was doing, so never point two concurrent Kestra executions at the same instanceName. ComputeEngine never deletes an instance it did not create, regardless of deleteInstance; use stopInstance if you also want it stopped.

Uses a GCS bucket for staging inputFiles/namespaceFiles and collecting outputFiles (required when using those features). The instance's own network must be able to reach Cloud Storage and the metadata server (an external IP, Cloud NAT, or Private Google Access all work) and its service account needs at least the https://www.googleapis.com/auth/devstorage.read_write scope for that bucket.

To access the task's working directory, use the {{workingDir}} Pebble expression or the WORKING_DIR environment variable — unlike the Batch and Cloud Run runners, commands here already execute from that directory. Output files can be created with the same name as listed in outputFiles anywhere in that directory, or under {{outputDir}} (OUTPUT_DIR) for automatic capture.

If the Kestra Worker running this task is terminated, the instance keeps running until completion; when resume is true (default) the Worker reattaches to it after restarting instead of creating a duplicate.

yaml
type: io.kestra.plugin.ee.gcp.runner.ComputeEngine

Create a Compute Engine VM, run a Shell command, then delete the VM.

yaml
id: compute_engine_shell
namespace: company.team

variables:
  projectId: "my-project"
  zone: "europe-west1-b"

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.ComputeEngine
      projectId: "{{ vars.projectId }}"
      zone: "{{ vars.zone }}"
      serviceAccount: "{{ secret('GOOGLE_SA') }}"
      instanceConfig:
        disks:
          - boot: true
            initializeParams:
              sourceImage: "projects/debian-cloud/global/images/family/debian-12"
        networkInterfaces:
          - network: "projects/{{ vars.projectId }}/global/networks/default"
            accessConfigs:
              - type: "ONE_TO_ONE_NAT"
    commands:
      - echo "Hello World"

Pass input files to an existing named instance, execute a Shell command, then retrieve output files.

yaml
id: compute_engine_existing_instance
namespace: company.team

inputs:
  - id: file
    type: FILE

variables:
  projectId: "my-project"
  zone: "europe-west1-b"
  bucket: "my-bucket"

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    inputFiles:
      data.txt: "{{ inputs.file }}"
    outputFiles:
      - out.txt
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.ComputeEngine
      projectId: "{{ vars.projectId }}"
      zone: "{{ vars.zone }}"
      bucket: "{{ vars.bucket }}"
      instanceName: "my-persistent-vm"
      stopInstance: false
      deleteInstance: false
      serviceAccount: "{{ secret('GOOGLE_SA') }}"
    commands:
      - cp {{ workingDir }}/data.txt {{ workingDir }}/out.txt
Properties

GCP zone

Zone the Compute Engine instance runs in, for example europe-west1-b.

Staging GCS bucket

Bucket used to upload input/namespace files and retrieve outputs; required when using file transfer or {{outputDir}}.

DefaultPT5S

Completion poll interval

How often to check the completion guest attribute and the serial console for new log lines; defaults to PT5S. Setting this property to a lower value will reduce the number of API calls Kestra makes to the remote service — keep that in mind in case you see API rate limit errors.

Defaulttrue

Delete the instance after completion

Defaults to true; set false to inspect or resume but stale instances may be reused. Never deletes an instance targeted through instanceName — ComputeEngine never deletes an instance it did not create — but it still controls whether that run's staged GCS blob prefix is cleaned up, regardless of instance ownership.

The GCP service account to impersonate

Service account email to impersonate for API calls. For Cloud Run runner, this value applies to API calls used to create and run the job (--impersonate-service-account equivalent). It does not set the job execution identity (--service-account).

New instance configuration

Raw JSON object used to create a new instance, in the same shape as the instances.insert request body — at minimum disks (with initializeParams.sourceImage) and networkInterfaces. Required unless instanceName is set. Any name field is ignored: ComputeEngine always generates the instance name from the flow/task/execution. The instance's network must be able to reach Cloud Storage and the metadata server (an external IP, Cloud NAT, or Private Google Access all work).

Existing instance name

Name of an already existing instance to run the task on, instead of creating a new one. Since a metadata startup-script only runs on boot, ComputeEngine reboots (instances.reset) the instance to (re)trigger it if it is already running when the task starts — this is destructive to anything else the instance is doing. Never target the same instanceName from two concurrent Kestra executions: the second run's reboot kills the first run's in-flight script. When set, instanceConfig is ignored. deleteInstance never deletes an instance targeted through instanceName — ComputeEngine never deletes an instance it did not create — but it still controls whether that run's staged GCS blob prefix is cleaned up, regardless of instance ownership.

Defaulte2-medium

Compute machine type

VM type for a newly created instance; defaults to e2-medium. Always overrides any machineType set inside instanceConfig. Only applies when creating a new instance. See https://cloud.google.com/compute/docs/machine-types

Reference (ref) of the pluginDefaults to apply to this task runner.

The GCP project ID

Defaulttrue

Resume an existing instance

If true (default) and instanceName is not set, reattach to a matching Kestra-created instance instead of creating a new one — used when the Worker restarts mid-run.

SubTypestring
Default["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used

The GCP service account key

Service account JSON key used to authenticate API calls. For Cloud Run runner job execution identity, this value is used as a fallback for --service-account when runtimeServiceAccount is not provided.

Defaulttrue

Stop the instance after completion

Defaults to true. Ignored if deleteInstance is also true (deleting supersedes stopping), and never applies to an instance targeted through instanceName.

Defaultfalse

Whether to synchronize working directory from remote runner back to local one after run.

Plugin Version

Defines the version of the plugin to use.

The version must follow the Semantic Versioning (SemVer) specification:

  • A single-digit MAJOR version (e.g., 1).
  • A MAJOR.MINOR version (e.g., 1.1).
  • A MAJOR.MINOR.PATCH version, optionally with any qualifier (e.g., 1.1.2, 1.1.0-SNAPSHOT).
DefaultPT5S

Post-completion log wait

Quiet period after the task ends: Kestra keeps polling the serial console for new log lines until none have arrived for this long, then finalizes logs and outputs; defaults to PT5S.

DefaultPT1H

Completion timeout

Maximum wall-clock duration to wait for the task script to complete before timing out; defaults to PT1H. Task timeout takes precedence.

More Plugins in this Category

Core Plugins and tasks
Core Plugins and tasks

Tasks that provide Kestra's built-in orchestration, I/O, and observability capabilities.

Core

122 tasks

398 blueprints

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks