Airbyte Cloud Jobs Sync

Airbyte Cloud Jobs Sync

Certified

Run an Airbyte Cloud sync

Starts a sync job for an Airbyte Cloud connection. By default, the task waits for completion and emits sync metrics.

The task also exposes the connection that ran the job, a connectionSnapshot file (connector configurations are never included) and a jobUrl to the job in Airbyte Cloud. When Kestra runs with assets enabled (Enterprise Edition) and the job succeeds, each replicated stream is emitted as a table asset with the source stream as its input. Airbyte Cloud does not report per-stream statistics, so the asset id carries no namespace unless the connection uses a custom namespace format.

Breaking change: this task now fails when the Airbyte job ends FAILED or CANCELLED. The outputs stay available: use allowFailure: true or an errors branch to keep the previous behavior.

Per-stream and per-attempt statistics, the attempts output and onPartialSuccess are not available on Airbyte Cloud, because its public API only reports bytesSynced, rowsSynced and duration for a job.

yaml
type: io.kestra.plugin.airbyte.cloud.jobs.Sync
yaml
id: airbyte_sync
namespace: company.team

tasks:
  - id: sync
    type: io.kestra.plugin.airbyte.cloud.jobs.Sync
    token: "{{ secret('AIRBYTE_TOKEN') }}"
    connectionId: e3b1ce92-547c-436f-b1e8-23b6936c12cd

Link to the Airbyte Cloud job when the sync fails

yaml
id: airbyte_cloud_sync_with_link
namespace: company.team

tasks:
  - id: sync
    type: io.kestra.plugin.airbyte.cloud.jobs.Sync
    token: "{{ secret('AIRBYTE_TOKEN') }}"
    connectionId: e3b1ce92-547c-436f-b1e8-23b6936c12cd

errors:
  - id: alert
    type: io.kestra.plugin.core.log.Log
    level: ERROR
    message: "Airbyte Cloud sync failed, see {{ outputs.sync.jobUrl }}"
Properties

Connection ID

Airbyte Cloud connection ID for the job

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions
assetFailureBehaviorstring
Possible Values
IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

id*string
Min length1
typestring
outputs

The assets produced as outputs.

id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*string
Min length1

Custom asset type

descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150

Client ID

Airbyte application client ID used for client-credentials authentication

Client secret

Airbyte application client secret used for client-credentials authentication. Store this value in a Secret

DefaultPT1H

Maximum wait duration

Maximum total time to wait for the job to finish, including Airbyte's automatic retry attempts. Defaults to 60 minutes

Basic auth password

Password for Airbyte Cloud basic authentication. Store this value in a Secret

DefaultPT1S

Poll frequency

Interval between Airbyte Cloud job status checks. Defaults to 1 second

Bearer token

Bearer token for the Airbyte Cloud API. If this is set, it is used before client credentials or basic auth

Defaulthttps://api.airbyte.com/v1/applications/token

Token URL

Token endpoint used with client credentials. Defaults to https://api.airbyte.com/v1/applications/token. See: https://reference.airbyte.com/reference/createaccesstoken

Basic auth username

Username for Airbyte Cloud basic authentication. Basic auth is used only when no bearer token or client credentials are configured

Defaulttrue

Wait for completion

If true, wait for the Airbyte Cloud job to reach a terminal state before the task completes. Defaults to true

Connection

Summary of the connection, source and destination of the job. null when their configuration could not be read

Definitions

Short, non-secret description of the connection that ran the job

connectionIdstring

Connection ID

destinationIdstring

Destination ID

destinationNamestring

Destination name

Name given to the destination in Airbyte

destinationTypestring

Destination type

Destination connector type, for example Snowflake

namestring

Name

scheduleTypestring

Schedule type

sourceIdstring

Source ID

sourceNamestring

Source name

Name given to the source in Airbyte

sourceTypestring

Source type

Source connector type, for example Postgres

statusstring

Status

streamCountinteger

Selected stream count

Number of streams replicated by the connection

workspaceIdstring

Workspace ID

Formaturi

Connection snapshot file

Internal storage URI of a JSON file with the connection, source and destination configuration at the time of the job. Connector configurations are never included. null when it could not be read

Job details

Airbyte Cloud job returned by the task

Definitions
bytesSyncedinteger

Bytes synced

Number of bytes synced by the job

durationstring
Formatduration

Duration

Total duration of the job

jobIdinteger

Job ID

Airbyte Cloud job ID

jobTypestring
Possible Values
SYNCRESETREFRESHCLEAR

Job type

Type of job (e.g. SYNC or RESET)

lastUpdatedAtstring
Formatdate-time

Last updated at

Timestamp when the job was last updated

rowsSyncedinteger

Rows synced

Number of rows synced by the job

startTimestring
Formatdate-time

Start time

Timestamp when the job started

statusstring
Possible Values
PENDINGRUNNINGINCOMPLETEFAILEDSUCCEEDEDCANCELLED

Status

Job status. Terminal when wait is enabled

Job URL

Link to the job in the Airbyte Cloud web app. null when the workspace could not be resolved

More Plugins in this Category

Firecrawl
Firecrawl

Tasks that scrape and crawl websites into clean, LLM-ready content with Firecrawl.

Data

2 tasks

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks

8 blueprints