Run Kestra Tasks on Google Compute Engine VMs

Table of Contents

For the complete documentation index, see llms.txt. For a full content snapshot, see llms-full.txt. Append .md to any kestra.io/docs/* URL for plain Markdown.

Available on:Enterprise Edition Cloud

Run script tasks directly on a Compute Engine VM instance, with no SSH or IAP tunnel required.

Overview

The Google Compute Engine task runner injects the task script as the instance’s startup-script metadata. When the VM boots, the guest environment runs the script automatically, mirrors its stdout and stderr to the serial console (port 1), and Kestra streams those lines back as task logs. Completion is detected via a kestra/status guest attribute the wrapper script writes on exit.

This mechanism requires:

Guest attributes enabled: the runner sets enable-guest-attributes automatically on every instance it touches.
An image with bash, curl, and python3: the default Debian and Ubuntu Compute Engine images satisfy this. Custom images must include all three.

Unlike the Google Batch and Google Cloud Run runners, the Compute Engine runner executes commands from the working directory. Use {{ workingDir }} or the WORKING_DIR environment variable when you need the explicit path.

If your project enforces the compute.disableGuestAttributesAccess organization policy, the runner fails fast with an actionable error on the first poll. Guest attributes cannot be selectively enabled for a single instance when this org policy is active.

Instance modes

Create a new instance (default)

Provide instanceConfig — a raw JSON object matching the instances.insert request body. At minimum, include disks (with initializeParams.sourceImage) and networkInterfaces. Any name field is ignored; Kestra generates the instance name from the flow, task, and execution.

taskRunner:
  type: io.kestra.plugin.ee.gcp.runner.ComputeEngine
  projectId: "{{ secret('GCP_PROJECT_ID') }}"
  zone: europe-west1-b
  serviceAccount: "{{ secret('GOOGLE_SA') }}"
  instanceConfig:
    disks:
      - boot: true
        initializeParams:
          sourceImage: projects/debian-cloud/global/images/family/debian-12
    networkInterfaces:
      - network: projects/my-project/global/networks/default
        subnetwork: projects/my-project/regions/europe-west1/subnetworks/default
    shieldedInstanceConfig:
      enableSecureBoot: true
      enableVtpm: true
      enableIntegrityMonitoring: true
    serviceAccounts:
      - email: default
        scopes:
          - https://www.googleapis.com/auth/devstorage.read_write

The machineType property (default e2-medium) always overrides any machineType set inside instanceConfig.

Target an existing instance

Set instanceName to run the task on an already-existing VM. Startup scripts run only on boot, so the runner reboots the instance (instances.reset) if it is already running.

Rebooting an existing instance interrupts anything else currently running on it. Never target the same instanceName from two concurrent Kestra executions. The second reboot kills the first task’s in-flight script.

The runner never deletes an instance it did not create. Setting deleteInstance: true when using instanceName only cleans up the staged GCS blob prefix for that run; the instance itself is left untouched. stopInstance has no effect on instances targeted via instanceName. Stop the instance manually when you are done with it.

taskRunner:
  type: io.kestra.plugin.ee.gcp.runner.ComputeEngine
  projectId: "{{ secret('GCP_PROJECT_ID') }}"
  zone: europe-west1-b
  serviceAccount: "{{ secret('GOOGLE_SA') }}"
  instanceName: my-persistent-vm
  stopInstance: false
  deleteInstance: false

File handling

Set bucket to enable inputFiles, namespaceFiles, outputFiles, and {{ outputDir }}. The bucket acts as a staging area:

Input and namespace files are uploaded before the instance starts the task.
Output files are collected from the instance after the task completes and made available in the Kestra UI.

bucket is required when any file transfer is configured. The runner raises an error at task start if output files are requested without a bucket.

The instance’s network must be able to reach Cloud Storage and the metadata server. Enable Private Google Access on the subnet or use Cloud NAT. Both let the VM reach storage.googleapis.com without a public IP. An external IP also works but is not recommended; see Security hardening.

When creating a new instance via instanceConfig, include serviceAccounts with at least the https://www.googleapis.com/auth/devstorage.read_write scope. The startup script fetches a GCS token from http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token; without an attached service account, this call fails and file transfer silently breaks.

The runner’s scopes property controls the OAuth scopes for Kestra’s own GCP credentials (used to call the Compute Engine and Cloud Storage APIs). It has no effect on the VM’s service account scopes. Set those in instanceConfig.serviceAccounts[].scopes.

taskRunner:
  type: io.kestra.plugin.ee.gcp.runner.ComputeEngine
  projectId: "{{ secret('GCP_PROJECT_ID') }}"
  zone: europe-west1-b
  bucket: my-staging-bucket
  serviceAccount: "{{ secret('GOOGLE_SA') }}"
  instanceConfig:
    disks:
      - boot: true
        initializeParams:
          sourceImage: projects/debian-cloud/global/images/family/debian-12
    networkInterfaces:
      - network: projects/my-project/global/networks/default
        subnetwork: projects/my-project/regions/europe-west1/subnetworks/default
    serviceAccounts:
      - email: default
        scopes:
          - https://www.googleapis.com/auth/devstorage.read_write

Security hardening

A common instanceConfig pattern includes accessConfigs: [{type: ONE_TO_ONE_NAT}], which gives the VM a public IP. Security Command Center flags this as a high-severity finding. The recommended setup omits the external IP and uses Private Google Access or Cloud NAT instead.

No external IP

Omit accessConfigs from networkInterfaces and specify subnetwork explicitly. The metadata server is always reachable from inside the VPC, so tasks that don’t use a bucket need no other connectivity. Tasks that do need to reach storage.googleapis.com — enable Private Google Access on the subnet or use Cloud NAT:

networkInterfaces:
  - network: projects/my-project/global/networks/default
    subnetwork: projects/my-project/regions/europe-west1/subnetworks/default
    # No accessConfigs — no external IP

If GCS input or output fails after removing the external IP, check that Private Google Access is enabled on the subnet, or that a Cloud NAT gateway covers it.

Images that need internet access at boot time (for example, to install packages) still need Cloud NAT. Private Google Access only covers Google API endpoints, not the general internet.

Shielded VM

Enable Secure Boot, vTPM, and integrity monitoring via shieldedInstanceConfig. The standard Debian and Ubuntu Compute Engine images support all three:

shieldedInstanceConfig:
  enableSecureBoot: true
  enableVtpm: true
  enableIntegrityMonitoring: true

Custom or older images may not support Shielded VM. The instances.insert call fails if the image does not support the requested shielded features.

Least-privilege VM scope

Grant the VM only the devstorage.read_write scope instead of cloud-platform. This is sufficient for all GCS file transfer the runner performs:

serviceAccounts:
  - email: default
    scopes:
      - https://www.googleapis.com/auth/devstorage.read_write

Hardened example

taskRunner:
  type: io.kestra.plugin.ee.gcp.runner.ComputeEngine
  projectId: "{{ secret('GCP_PROJECT_ID') }}"
  zone: europe-west1-b
  bucket: my-staging-bucket
  serviceAccount: "{{ secret('GOOGLE_SA') }}"
  instanceConfig:
    disks:
      - boot: true
        initializeParams:
          sourceImage: projects/debian-cloud/global/images/family/debian-12
    networkInterfaces:
      - network: projects/my-project/global/networks/default
        subnetwork: projects/my-project/regions/europe-west1/subnetworks/default
    shieldedInstanceConfig:
      enableSecureBoot: true
      enableVtpm: true
      enableIntegrityMonitoring: true
    serviceAccounts:
      - email: default
        scopes:
          - https://www.googleapis.com/auth/devstorage.read_write

IAM permissions

Grant the following IAM roles to the service account Kestra uses (serviceAccount):

Role	Required for
`roles/compute.instanceAdmin.v1`	Create, start, stop, delete instances; read guest attributes; read the serial console
`roles/iam.serviceAccountUser`	Required when the Compute Engine instance runs under a dedicated service account
`roles/storage.admin`	Required when using `inputFiles`, `outputFiles`, or `namespaceFiles` with a `bucket`

Instance lifecycle

By default, the runner creates a new instance for every task run and deletes it on completion.

Property	Default	Behavior
`stopInstance`	`true`	Stop the instance after the task completes. Ignored when `deleteInstance` is also `true`. Never applies to instances targeted via `instanceName`.
`deleteInstance`	`true`	Delete the instance and clean up its staged GCS prefix after the task completes. Never deletes an instance targeted via `instanceName`.

Set both to false to keep the instance running for inspection after the task:

taskRunner:
  type: io.kestra.plugin.ee.gcp.runner.ComputeEngine
  projectId: "{{ secret('GCP_PROJECT_ID') }}"
  zone: europe-west1-b
  serviceAccount: "{{ secret('GOOGLE_SA') }}"
  stopInstance: false
  deleteInstance: false
  instanceConfig:
    disks:
      - boot: true
        initializeParams:
          sourceImage: projects/debian-cloud/global/images/family/debian-12
    networkInterfaces:
      - network: projects/my-project/global/networks/default

Termination and resume behavior

If the Kestra Worker is terminated mid-run, the instance continues running until the task script finishes. When resume: true (default), a restarted Worker reattaches to the existing Kestra-labeled instance rather than creating a duplicate.

If you stop the execution from the Kestra UI, Kestra deletes any instance it created for that run. Instances targeted via instanceName are left running with a log warning.

Set resume: false to force a new instance on every execution attempt:

taskRunner:
  type: io.kestra.plugin.ee.gcp.runner.ComputeEngine
  projectId: "{{ secret('GCP_PROJECT_ID') }}"
  zone: europe-west1-b
  serviceAccount: "{{ secret('GOOGLE_SA') }}"
  resume: false
  instanceConfig:
    disks:
      - boot: true
        initializeParams:
          sourceImage: projects/debian-cloud/global/images/family/debian-12
    networkInterfaces:
      - network: projects/my-project/global/networks/default

Timeout and polling

Property	Default	Description
`waitUntilCompletion`	`PT1H`	Maximum wall-clock time before the task times out. The task’s own `timeout` takes precedence when set.
`completionCheckInterval`	`PT5S`	How often to poll the guest attribute and stream new serial console log lines.
`waitForLogInterval`	`PT5S`	Quiet period after the task ends. Kestra keeps polling the serial console until no new log lines arrive for this duration, then finalizes logs and outputs.

taskRunner:
  type: io.kestra.plugin.ee.gcp.runner.ComputeEngine
  projectId: "{{ secret('GCP_PROJECT_ID') }}"
  zone: europe-west1-b
  serviceAccount: "{{ secret('GOOGLE_SA') }}"
  waitUntilCompletion: PT4H
  completionCheckInterval: PT30S
  waitForLogInterval: PT10S
  instanceConfig:
    disks:
      - boot: true
        initializeParams:
          sourceImage: projects/debian-cloud/global/images/family/debian-12
    networkInterfaces:
      - network: projects/my-project/global/networks/default

Example flows

Create a VM and run a shell command

id: compute_engine_shell
namespace: company.team

variables:
  projectId: my-project
  zone: europe-west1-b

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.ComputeEngine
      projectId: "{{ vars.projectId }}"
      zone: "{{ vars.zone }}"
      serviceAccount: "{{ secret('GOOGLE_SA') }}"
      instanceConfig:
        disks:
          - boot: true
            initializeParams:
              sourceImage: projects/debian-cloud/global/images/family/debian-12
        networkInterfaces:
          - network: "projects/{{ vars.projectId }}/global/networks/default"
            subnetwork: "projects/{{ vars.projectId }}/regions/europe-west1/subnetworks/default"
        shieldedInstanceConfig:
          enableSecureBoot: true
          enableVtpm: true
          enableIntegrityMonitoring: true
        serviceAccounts:
          - email: default
            scopes:
              - https://www.googleapis.com/auth/devstorage.read_write
    commands:
      - echo "Hello World"

Pass input files to an existing instance and retrieve output files

id: compute_engine_existing_instance
namespace: company.team

inputs:
  - id: file
    type: FILE

variables:
  projectId: my-project
  zone: europe-west1-b
  bucket: my-bucket

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    inputFiles:
      data.txt: "{{ inputs.file }}"
    outputFiles:
      - out.txt
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.ComputeEngine
      projectId: "{{ vars.projectId }}"
      zone: "{{ vars.zone }}"
      bucket: "{{ vars.bucket }}"
      instanceName: my-persistent-vm
      stopInstance: false
      deleteInstance: false
      serviceAccount: "{{ secret('GOOGLE_SA') }}"
    commands:
      - cp {{ workingDir }}/data.txt {{ workingDir }}/out.txt

For a complete list of properties, see the plugin documentation or explore the configuration in the built-in Code Editor.

Google Cloud Run Task RunnerGoogle Cloud Run Task Runner: Serverless Task Execution

Was this page helpful?