Google Cloud WaitForJob
CertifiedWait for a Dataflow job to complete
Google Cloud WaitForJob
Certified
Wait for a Dataflow job to complete
Polls a Dataflow job status sequentially until it enters a terminal state (JOB_STATE_DONE, JOB_STATE_FAILED, JOB_STATE_CANCELLED, JOB_STATE_DRAINED, JOB_STATE_UPDATED).
yaml
type: io.kestra.plugin.gcp.dataflow.WaitForJobExamples
Launch a template job and wait for its completion.
yaml
id: dataflow_classic_template
namespace: company.team
inputs:
- id: input_path
type: STRING
tasks:
- id: launch
type: io.kestra.plugin.gcp.dataflow.LaunchTemplate
projectId: "{{ secret('GCP_PROJECT_ID') }}"
location: us-central1
jobName: my-etl-job
gcsPath: gs://my-bucket/templates/my-template
parameters:
inputFile: "{{ inputs.input_path }}"
outputTable: my-project:my_dataset.my_table
- id: wait
type: io.kestra.plugin.gcp.dataflow.WaitForJob
projectId: "{{ secret('GCP_PROJECT_ID') }}"
location: us-central1
jobId: "{{ outputs.launch.jobId }}"
pollInterval: PT15S
maxDuration: PT1H
Properties
jobId *Requiredstring
The Dataflow job ID
location *Requiredstring
The regional endpoint (e.g. us-central1)
impersonatedServiceAccount string
The GCP service account to impersonate
maxDuration string
Default
PT1HThe maximum duration to wait before timing out
pluginDefaultsRef Non-dynamicstring
Reference (ref) of the pluginDefaults to apply to this task.
pollInterval string
Default
PT15SThe interval between polls
projectId string
The GCP project ID
scopes array
SubTypestring
Default
["https://www.googleapis.com/auth/cloud-platform"]The GCP scopes to be used
serviceAccount string
The GCP service account
Outputs
jobId string
The job ID
metrics object
A map containing job metrics at completion
state string
The final terminal state of the job
