Google Cloud WaitForJob
CertifiedWait for a Dataflow job to complete
Google Cloud WaitForJob
Certified
Wait for a Dataflow job to complete
Polls a Dataflow job status sequentially until it enters a terminal state (JOB_STATE_DONE, JOB_STATE_FAILED, JOB_STATE_CANCELLED, JOB_STATE_DRAINED, JOB_STATE_UPDATED).
yaml
type: io.kestra.plugin.gcp.dataflow.WaitForJobExamples
Launch a template job and wait for its completion.
yaml
id: dataflow_classic_template
namespace: company.team
inputs:
- id: input_path
type: STRING
tasks:
- id: launch
type: io.kestra.plugin.gcp.dataflow.LaunchTemplate
projectId: "{{ secret('GCP_PROJECT_ID') }}"
location: us-central1
jobName: my-etl-job
gcsPath: gs://my-bucket/templates/my-template
parameters:
inputFile: "{{ inputs.input_path }}"
outputTable: my-project:my_dataset.my_table
- id: wait
type: io.kestra.plugin.gcp.dataflow.WaitForJob
projectId: "{{ secret('GCP_PROJECT_ID') }}"
location: us-central1
jobId: "{{ outputs.launch.jobId }}"
pollInterval: PT15S
maxDuration: PT1H
Properties
jobId *Requiredstring
The Dataflow job ID
location *Requiredstring
The regional endpoint (e.g. us-central1)
impersonatedServiceAccount string
The GCP service account to impersonate
maxDuration string
Default
PT1HThe maximum duration to wait before timing out
pollInterval string
Default
PT15SThe interval between polls
projectId string
The GCP project ID
scopes array
SubTypestring
Default
["https://www.googleapis.com/auth/cloud-platform"]The GCP scopes to be used
serviceAccount string
The GCP service account
Outputs
jobId string
The job ID
metrics object
A map containing job metrics at completion
state string
The final terminal state of the job
