Airbyte Async Sync Completion Verification Gate

Not every Airbyte sync is triggered and awaited in the same flow; a sync kicked off asynchronously, from Airbyte's own UI, a webhook, or a separate Kestra flow that does not wait, leaves nothing watching for it to actually finish. This blueprint takes just the job ID and confirms, independently, whether that job reached SUCCEEDED, CANCELLED, or FAILED, since a job that never confirms success is exactly the kind of silent gap a governance check exists to catch.

This blueprint was created by Mohit Jeswani.

How it works

1. check_status (io.kestra.plugin.airbyte.connections.CheckStatus) polls job_id every second until it reaches a terminal state, for up to max_wait_minutes. It only returns normally when the job's final status is SUCCEEDED; any other terminal state, or a timeout, makes the task throw, which allowFailure: true turns into an undefined output here instead of a hard flow failure.
2. evaluate (io.kestra.plugin.core.output.OutputValues) classifies an undefined check_status output as SYNC_NOT_CONFIRMED, otherwise OK, and reads the real rowsSynced count when available.
3. route (io.kestra.plugin.core.flow.If):
   • compliant: logs the confirmed success and rows synced.
   • alert: posts the job ID and wait window to Slack.
4. summary logs the final severity, closing the governance cycle.
5. errors alerts Slack if the flow itself fails outright, for example an unreachable Airbyte instance.

What you get

• An independent confirmation that an asynchronously-triggered sync actually succeeded, instead of assuming silence means success.
• The real rowsSynced count in the compliant log, not just a pass/fail flag.
• A read-only check against the job: this flow never re-triggers the sync or cancels anything in Airbyte.

Who it's for

• Data platform teams running Airbyte who trigger syncs outside a single waiting Kestra flow (from Airbyte's own UI, a webhook, or a fire-and-forget upstream task) and need confirmation those syncs actually finished.
• Teams evaluating the Airbyte plugin who want a worked example of connections.CheckStatus used as a standing verification gate for an async job id, distinct from this repo's existing Airbyte blueprints, which trigger and wait for a sync within one task rather than verifying a job id captured elsewhere.

Why orchestrate this with Kestra

Airbyte's own API answers "what is this job's status" on request; it does not keep polling on your behalf, decide a timeout, or notify a Slack channel if the job never confirms success. Kestra keeps max_wait_minutes as a flow input, turns an unconfirmed sync into a Slack page naming the exact job id, and keeps a full execution history of every check.

Prerequisites

• A running Airbyte instance (the official airbyte/airbyte Docker Compose stack works for local testing) with at least one connection you can trigger a sync against.
• A real Airbyte job id to verify, from a prior Sync task run with wait: false, or from Airbyte's own UI.
• A Slack incoming webhook URL.

Secrets

• SLACK_WEBHOOK_URL: used by alert and the errors block. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_SLACK_WEBHOOK_URL.

Inputs

Input	Default	Purpose
airbyte_url	http://localhost:8080	Base URL of the Airbyte instance
job_id	<your-airbyte-job-id>	Job id to verify
max_wait_minutes	30	How long to keep polling before treating the job as timed out

Outputs

• {{ outputs.check_status.finalJobStatus }} / .metadata.rowsSynced: only populated when the job confirmed SUCCEEDED.
• {{ outputs.evaluate.values.rows_synced }} / .severity: the rows synced (or -1 when unconfirmed) and OK or SYNC_NOT_CONFIRMED.

Pitfalls

• check_status throws rather than returning a row for FAILED, CANCELLED, or a timeout; this blueprint relies on allowFailure: true to turn that into a classifiable SYNC_NOT_CONFIRMED state rather than aborting the flow, the real failure reason is in the task's own logs, not in evaluate's output.
• job_id has to come from somewhere outside this flow; there is no task here that discovers running jobs on its own, this blueprint only verifies a job id you already have.
• maxDuration here bounds how long check_status itself polls; a job that is still genuinely running past that window is reported SYNC_NOT_CONFIRMED even though it might still succeed moments later. Raise max_wait_minutes for a connection with long sync times.

Quick start

1. Add the SLACK_WEBHOOK_URL secret and point airbyte_url at a running Airbyte instance.
2. Trigger a real sync with a Sync task set to wait: false, note the returned jobId.
3. Set job_id to that value and run this flow; confirm compliant logs the rows synced once the job finishes.
4. Point job_id at a job you know will fail (or a bogus id) to confirm alert fires as SYNC_NOT_CONFIRMED.

How to extend

• Chain this flow directly after a Sync task set to wait: false, passing its jobId output straight into job_id, for a two-flow fire-and-verify pattern.
• Persist job_id to a KV entry from the triggering flow and have the hourly trigger read the most recent pending job id instead of a fixed input.
• Add io.kestra.plugin.airbyte.connections.Sync with onActiveSync: ADOPT in the alert branch to retry the sync, gated behind an explicit confirm_retry input.

Links

• Airbyte plugin: https://kestra.io/plugins/plugin-airbyte
• Airbyte Docker Compose quickstart: https://docs.airbyte.com/platform/using-airbyte/getting-started/oss-quickstart
• Slack incoming webhooks: https://api.slack.com/messaging/webhooks