Google Cloud Batch

Google Cloud Batch

Certified
Enterprise Edition

Run tasks on Google Cloud Batch

Enterprise-only task runner that launches the container image as a Cloud Batch Job; requires Batch Jobs Editor and Logs Viewer roles. Uses a GCS bucket for staging input/namespace files and collecting outputs (required when using those features), polls every 5s by default, and times out after 1h unless the task timeout overrides it. Jobs default to deleting on completion and resuming existing jobs when labels match; Cloud Logging captures stdout/stderr and the worker resumes a running job if the worker restarts. This task runner is container-based so the containerImage property must be set. You need to have roles 'Batch Job Editor' (roles/batch.jobsEditor) and 'Logs Viewer' (roles/logging.viewer) to be able to use it.

To access the task's working directory, use the {{workingDir}} Pebble expression or the WORKING_DIR environment variable. Input files and namespace files will be available in this directory.

To generate output files you can either use the outputFiles task's property and create a file with the same name in the task's working directory, or create any file in the output directory which can be accessed by the {{outputDir}} Pebble expression or the OUTPUT_DIR environment variables.

To use inputFiles, outputFiles or namespaceFiles properties, make sure to set the bucket property. The bucket serves as an intermediary storage layer for the task runner. Input and namespace files will be uploaded to the cloud storage bucket before the task run. Similarly, the task runner will store outputFiles in this bucket during the task run. In the end, the task runner will make those files available for download and preview from the UI by sending them to internal storage.

The task runner will generate a folder in the configured bucket for each task run. You can access that folder using the {{bucketPath}} Pebble expression or the BUCKET_PATH environment variable.

Warning, contrarily to other task runners, this task runner didn't run the task in the working directory but in the root directory. You must use the {{workingDir}} Pebble expression or the WORKING_DIR environment variable to access files.

Note that when the Kestra Worker running this task is terminated, the batch job will still runs until completion, then after restarting, the Worker will resume processing on the existing job unless resume is set to false.

yaml
type: io.kestra.plugin.ee.gcp.runner.Batch

Execute a Shell command.

yaml
id: new_shell
namespace: company.team

variables:
  projectId: "myproject"
  region: "europe-west2"

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.Batch
      projectId: "{{vars.projectId}}"
      region: "{{ vars.region} }"
      serviceAccount: "{{ secret('GOOGLE_SA') }}"
    commands:
      - echo "Hello World"

Pass input files to the task, execute a Shell command, then retrieve output files.

yaml
id: new_shell_with_file
namespace: company.team

inputs:
  - id: file
    type: FILE

variables:
  projectId: "myProject"
  region: "europe-west2"
  bucket: "myBucket"

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    inputFiles:
      data.txt: "{{ inputs.file }}"
    outputFiles:
      - out.txt
    containerImage: centos
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.Batch
      projectId: "{{ vars.projectId }}"
      region: "{{ vars.region }}"
      bucket: "{{ vars.bucket }}"
      serviceAccount: "{{ secret('GOOGLE_SA') }}"
    commands:
      - cp {{workingDir}}/data.txt {{workingDir}}/out.txt

Run a Python script to fetch environment information on Google Cloud with Google Batch

yaml
id: gcp_batch_runner
namespace: company.team

tasks:
  - id: scrape_environment_info
    type: io.kestra.plugin.scripts.python.Commands
    containerImage: ghcr.io/kestra-io/pydata:latest
    taskRunner:
      type: io.kestra.plugin.ee.gcp.runner.Batch
      projectId: "{{ secret('GCP_PROJECT_ID') }}"
      region: "europe-west9"
      bucket: "{{ secret('GCS_BUCKET')}}"
      serviceAccount: "{{ secret('GOOGLE_SA') }}"
    commands:
      - python {{ workingDir }}/main.py
    namespaceFiles:
      enabled: true
    outputFiles:
      - environment_info.json
    inputFiles:
      main.py: |
        import platform
        import socket
        import sys
        import json
        from kestra import Kestra

        print("Hello from GCP Batch and kestra!")

        def print_environment_info():
            print(f"Host's network name: {platform.node()}")
            print(f"Python version: {platform.python_version()}")
            print(f"Platform information (instance type): {platform.platform()}")
            print(f"OS/Arch: {sys.platform}/{platform.machine()}")

            env_info = {
                "host": platform.node(),
                "platform": platform.platform(),
                "OS": sys.platform,
                "python_version": platform.python_version(),
            }
            Kestra.outputs(env_info)

            filename = '{{ workingDir }}/environment_info.json'
            with open(filename, 'w') as json_file:
                json.dump(env_info, json_file, indent=4)

        if __name__ == '__main__':
          print_environment_info()
Properties

GCP region

Region where the Batch job runs.

Staging GCS bucket

Bucket used to upload input/namespace files and retrieve outputs; required when using file transfer or {{outputDir}}.

DefaultPT5S

Completion poll interval

How often to poll the job status, and the default cadence for log polling unless logPollInterval overrides it. Defaults to PT5S. Lower for short jobs, higher to reduce API calls.

Per-task compute resources

ComputeResource defines the amount of resources required for each task. Make sure your tasks have enough compute resources to successfully run. If you also define the types of resources for a job to use with the InstancePolicyOrTemplate field, make sure both fields are compatible with each other. Override CPU, memory, and boot disk per task (defaults: cpu 2000 milliCPU, memory 2048 MiB). Values must stay compatible with the chosen machine type or instance policy.

Definitions
bootDiskstring

Extra boot disk size

Additional boot disk size per task (e.g., 10GiB).

cpustring

CPU in milliCPU

Defines the amount of CPU resources per task in milliCPU units. For example, 1000 corresponds to 1 vCPU per task. If undefined, the default value is 2000. If you also define the VM's machine type using the machineType property in InstancePolicy or inside the instanceTemplate in InstancePolicyOrTemplate, make sure the CPU resources for both fields are compatible with each other and with how many tasks you want to allow to run on the same VM at the same time. For example, if you specify the n2-standard-2 machine type, which has 2 vCPUs, you can set the cpu to no more than 2000. Alternatively, you can run two tasks on the same VM if you set the cpu to 1000 or less.

memorystring

Memory in MiB

Per-task memory request in MiB; defaults to 2048. Must stay within the chosen machine type or instance template. If you also define the VM's machine type using the machineType in InstancePolicy or inside the instanceTemplate in InstancePolicyOrTemplate, make sure the memory resources for both fields are compatible with each other and with how many tasks you want to allow to run on the same VM at the same time. For example, if you specify the n2-standard-2 machine type, which has 8 GiB of memory, you can set the memory to no more than 8192.

Defaulttrue

Delete job after completion

Defaults to true; set false to inspect or resume but stale jobs may be reused.

SubTypestring

Container entrypoint

Override container entrypoint command.

The GCP service account to impersonate

Service account email to impersonate for API calls. For Cloud Run runner, this value applies to API calls used to create and run the job (--impersonate-service-account equivalent). It does not set the job execution identity (--service-account).

Lifecycle policy for failed tasks

Optional policy executed when conditions match; defaults to exit on code 0 and retry on non-zero up to max_retry_count.

Definitions
actionstring
Possible Values
ACTION_UNSPECIFIEDRETRY_TASKFAIL_TASKUNRECOGNIZED

Action on task failures

Batch action to take when lifecycle conditions are met.

actionCondition

Failure conditions

Conditions that trigger the lifecycle action (exit codes, etc.).

exitCodesarray
SubTypeinteger

Exit codes triggering the action

Lifecycle action fires when the task exits with any listed code.

Log poll interval

How often to poll Cloud Logging for new task log lines, independent of the job-status poll. Defaults to completionCheckInterval when unset. Cloud Logging's read path is capped at 60 requests per minute per project (a limit Google does not raise), while status polling has far more headroom, so raise this alone to relieve the log read quota at high concurrency without slowing completion detection.

Defaulte2-medium

Compute machine type

VM type for the Batch instance; defaults to e2-medium. See https://cloud.google.com/compute/docs/machine-types

Default2

Maximum number of retries when creating the batch job fails

Minimum>= 0
Maximum<= 10

Max task retries

Batch retry count on failure; default 0 disables retries.

Network interfaces

List of networks/subnets to attach to the Batch VM.

Definitions
network*Requiredstring

Network identifier with the format projects/HOST_PROJECT_ID/global/networks/NETWORK

subnetworkstring

Subnetwork identifier in the format projects/HOST_PROJECT_ID/regions/REGION/subnetworks/SUBNET

Reference (ref) of the pluginDefaults to apply to this task runner.

The GCP project ID

Compute reservation

Reservation resource name to target reserved capacity.

Defaulttrue

Resume existing job

If true (default), reattach to a matching job instead of creating a new one.

SubTypestring
Default["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used

The GCP service account key

Service account JSON key used to authenticate API calls. For Cloud Run runner job execution identity, this value is used as a fallback for --service-account when runtimeServiceAccount is not provided.

Defaultfalse

Whether to synchronize working directory from remote runner back to local one after run.

Plugin Version

Defines the version of the plugin to use.

The version must follow the Semantic Versioning (SemVer) specification:

  • A single-digit MAJOR version (e.g., 1).
  • A MAJOR.MINOR version (e.g., 1.1).
  • A MAJOR.MINOR.PATCH version, optionally with any qualifier (e.g., 1.1.2, 1.1.0-SNAPSHOT).
DefaultPT5S

Post-completion log wait

Quiet period after the job ends: Kestra keeps polling for new log entries until none have arrived for this long, then finalizes logs and outputs; defaults to PT5S.

DefaultPT1H

Job timeout

Maximum wall-clock duration before Batch times out the job; defaults to PT1H. Task timeout takes precedence.

Time spent draining late log entries after completion.

Log entries received from Cloud Logging.

Cloud Logging log polls issued, tagged with projectId and region.

Cloud Logging log polls that errored, the alertable leading indicator of read-quota exhaustion.

More Plugins in this Category

Core Plugins and tasks
Core Plugins and tasks

Tasks that provide Kestra's built-in orchestration, I/O, and observability capabilities.

Core

122 tasks

580 blueprints

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks

6 blueprints