Google Cloud CustomJob

Google Cloud CustomJob
Certified
Run a Vertex AI custom job

Starts a custom training job in Vertex AI with the provided spec. Can stream logs, wait for completion, and optionally delete the job afterward.

yaml
type: io.kestra.plugin.gcp.vertexai.CustomJob

Examples

yaml
id: gcp_vertexai_custom_job
namespace: company.team

tasks:
  - id: custom_job
    type: io.kestra.plugin.gcp.vertexai.CustomJob
    projectId: my-gcp-project
    region: europe-west1
    displayName: Start Custom Job
    spec:
      workerPoolSpecs:
      - containerSpec:
          imageUri: gcr.io/my-gcp-project/my-dir/my-image:latest
        machineSpec:
          machineType: n1-standard-4
        replicaCount: 1

Properties

displayName *string

Display name

Human-friendly name for the job

region *string

Region

Vertex AI region for the job endpoint

spec *

Job spec

CustomJobSpec defining worker pools, container/package, scheduling, etc.

Definitions

io.kestra.plugin.gcp.vertexai.models.CustomJobSpec

workerPoolSpecs*array

Min items1

Worker pool specs

At least one worker pool; define machine type, replica count, and container/package.

io.kestra.plugin.gcp.vertexai.models.WorkerPoolSpec

containerSpec*

The custom container task

io.kestra.plugin.gcp.vertexai.models.ContainerSpec

imageUri*string

The URI of a container image in the Container Registry that is to be run on each worker replica

Must be on google container registry, example: gcr.io/{{ project }}/{{ dir }}/{{ image }}: {{ tag }}

argsarray

SubTypestring

The arguments to be passed when starting the container

commandsarray

SubTypestring

The command to be invoked when the container is started

It overrides the entrypoint instruction in Dockerfile when provided.

envobject

Environment variables to be passed to the container

Maximum limit is 100.

machineSpec*

The specification of a single machine

io.kestra.plugin.gcp.vertexai.models.MachineSpec

machineType*string

The type of the machine

See the list of machine types supported forprediction See the list of machine types supported for custom training.

acceleratorCountintegerstring

The number of accelerators to attach to the machine

acceleratorTypestring

Possible Values

ACCELERATOR_TYPE_UNSPECIFIEDNVIDIA_TESLA_K80NVIDIA_TESLA_P100NVIDIA_TESLA_V100NVIDIA_TESLA_P4NVIDIA_TESLA_T4NVIDIA_TESLA_A100NVIDIA_A100_80GBNVIDIA_L4NVIDIA_H100_80GBNVIDIA_H100_MEGA_80GBNVIDIA_H200_141GBNVIDIA_B200NVIDIA_GB200NVIDIA_RTX_PRO_6000TPU_V2TPU_V3TPU_V4_PODTPU_V5_LITEPODUNRECOGNIZED

The type of accelerator(s) that may be attached to the machine

discSpec

The specification of the disk

io.kestra.plugin.gcp.vertexai.models.DiscSpec

bootDiskSizeGbintegerstring

Default100

Size in GB of the boot disk

bootDiskTypestring

DefaultPD_SSD

Possible Values

PD_SSDPD_STANDARD

Type of the boot disk

pythonPackageSpec

The python package specs

io.kestra.plugin.gcp.vertexai.models.PythonPackageSpec

args*array

SubTypestring

The Google Cloud Storage location of the Python package files which are the training program and its dependent packages

The maximum number of package URIs is 100.

envs*object

Environment variables to be passed to the python module

Maximum limit is 100.

packageUris*array

SubTypestring

The Google Cloud Storage location of the Python package files which are the training program and its dependent packages

The maximum number of package URIs is 100.

replicaCountintegerstring

The specification of the disk

baseOutputDirectory

Base output directory

GCS location for job outputs

io.kestra.plugin.gcp.vertexai.models.GcsDestination

outputUriPrefix*string

Google Cloud Storage URI to output directory

If the uri doesn't end with '/', a '/' will be automatically appended. The directory is created if it doesn't exist.

enableWebAccessbooleanstring

Enable web access

Enable interactive shell access to training containers

networkstring

VPC network

Full network name (projects/{project}/global/networks/{network}); requires Vertex AI VPC peering

scheduling

Scheduling

Optional scheduling options (timeout, restart, etc.)

io.kestra.plugin.gcp.vertexai.models.Scheduling

restartJobOnWorkerRestart*booleanstring

Restarts the entire CustomJob if a worker gets restarted

This feature can be used by distributed training jobs that are not resilient to workers leaving and joining a job.

timeOut*string

The maximum job running time. The default is 7 days

serviceAccountstring

Service account

Run-as service account; submitters need act-as on this account. Defaults to Vertex AI Custom Code Service Agent.

tensorboardstring

Tensorboard

Tensorboard resource for logs, format projects/{project}/locations/{location}/tensorboards/{tensorboard}

delete booleanstring

Defaulttrue

Delete on completion

If true (default), deletes the job after completion

impersonatedServiceAccount string

The GCP service account to impersonate

pluginDefaultsRef string

Reference (ref) of the pluginDefaults to apply to this task.

projectId string

The GCP project ID

scopes array

SubTypestring

Default["https://www.googleapis.com/auth/cloud-platform"]

The GCP scopes to be used

serviceAccount string

The GCP service account

wait booleanstring

Defaulttrue

Wait for completion

If true (default), waits for job end and captures status/logs

Outputs

createDate *string

Formatdate-time

Time when the CustomJob was created

endDate *string

Formatdate-time

Time when the CustomJob was ended

name *string

Resource name of a CustomJob

state *string

Possible Values

JOB_STATE_UNSPECIFIEDJOB_STATE_QUEUEDJOB_STATE_PENDINGJOB_STATE_RUNNINGJOB_STATE_SUCCEEDEDJOB_STATE_FAILEDJOB_STATE_CANCELLINGJOB_STATE_CANCELLEDJOB_STATE_PAUSEDJOB_STATE_EXPIREDJOB_STATE_UPDATINGJOB_STATE_PARTIALLY_SUCCEEDEDUNRECOGNIZED

The detailed state of the CustomJob