Google Cloud WaitForJob
CertifiedWait for a Dataflow job to complete
Google Cloud WaitForJob
Wait for a Dataflow job to complete
Polls a Dataflow job status sequentially until it enters a terminal state (JOB_STATE_DONE, JOB_STATE_FAILED, JOB_STATE_CANCELLED, JOB_STATE_DRAINED, JOB_STATE_UPDATED).
type: io.kestra.plugin.gcp.dataflow.WaitForJobExamples
Launch a template job and wait for its completion.
id: dataflow_classic_template
namespace: company.team
inputs:
- id: input_path
type: STRING
tasks:
- id: launch
type: io.kestra.plugin.gcp.dataflow.LaunchTemplate
projectId: "{{ secret('GCP_PROJECT_ID') }}"
location: us-central1
jobName: my-etl-job
gcsPath: gs://my-bucket/templates/my-template
parameters:
inputFile: "{{ inputs.input_path }}"
outputTable: my-project:my_dataset.my_table
- id: wait
type: io.kestra.plugin.gcp.dataflow.WaitForJob
projectId: "{{ secret('GCP_PROJECT_ID') }}"
location: us-central1
jobId: "{{ outputs.launch.jobId }}"
pollInterval: PT15S
maxDuration: PT1H
Properties
jobId *string
The Dataflow job ID
location *string
The regional endpoint (e.g. us-central1)
assets
Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.
io.kestra.core.models.assets.AssetsDeclaration
IGNOREFAILWARNAsset failure behavior
Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.
Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.
The assets consumed as inputs.
io.kestra.core.models.assets.AssetIdentifier
1The assets produced as outputs.
io.kestra.plugin.ee.assets.Dataset
1150{}1150io.kestra.plugin.ee.assets.File
1150{}1150io.kestra.plugin.ee.assets.Table
1150{}1150io.kestra.plugin.ee.assets.VM
1150{}1150io.kestra.core.models.assets.External
1150{}1150io.kestra.core.models.assets.Custom
11501Custom asset type
{}1150impersonatedServiceAccount string
The GCP service account to impersonate
maxDuration string
PT1HThe maximum duration to wait before timing out
pollInterval string
PT15SThe interval between polls
projectId string
The GCP project ID
scopes array
["https://www.googleapis.com/auth/cloud-platform"]The GCP scopes to be used
serviceAccount string
The GCP service account
Outputs
jobId string
The job ID
metrics object
A map containing job metrics at completion
state string
The final terminal state of the job
