How to use the Airbyte plugin

Trigger and monitor Airbyte syncs from Kestra flows — with separate task packages for self-hosted Airbyte and Airbyte Cloud.

Authentication

Self-hosted (connections.*): set url to your Airbyte instance URL. For authenticated instances, set username and password, or token for bearer token auth. For OAuth M2M, set applicationCredentials.clientId and applicationCredentials.clientSecret.

Airbyte Cloud (cloud.jobs.*): set clientId and clientSecret from your Airbyte Cloud workspace API credentials, or set token directly. Basic auth (username/password) is also supported as a fallback when neither is set.

Store secrets in secrets and set connection properties on each task.

Tasks

Self-hosted — connections.Sync triggers a sync by connectionId and waits for completion by default (wait: true). Completed syncs return metadata with rowsSynced and the affected source and destination tables, including their namespaces and per-table row counts. Metadata is null when wait: false or when an active sync is skipped because final job statistics are not yet available. If a sync is already running for the connection, onActiveSync controls what happens: ADOPT (default) attaches to the in-flight job and polls it to completion instead of triggering a new sync; FAIL fails the task immediately; SKIP succeeds immediately with alreadyRunning: true and a null jobId, without starting or queuing a second sync. failOnActiveSync is deprecated but still supported: when set, it overrides onActiveSync (true → FAIL, false → SKIP). Control polling with pollFrequency (default 1 second) and cap wait time with maxDuration (default 60 minutes) — for an adopted job, maxDuration is counted from the moment it is adopted, not from the job's actual start time. connections.CheckStatus polls an existing sync job by jobId until it reaches a terminal state and exposes the same metadata. connections.Reset resets one or more streams of a connection (streams, each with a streamName and an optional streamNamespace; at least one is required) through Airbyte's /api/v1/connections/reset/stream config API and returns the reset jobId. A reset deletes the data of the selected streams in the destination and resets their cursors, so the next sync replicates them from scratch. It does not wait for the job: pass the jobId to connections.CheckStatus to wait.

Job outputs and failures (self-hosted) — when wait is enabled, connections.Sync and connections.CheckStatus return job (status, timing, attempt counts, partialSuccess and sanitized failures), attempts (an internal storage Ion file with one row per attempt and its per-stream statistics) and an extended metadata (bytesSynced, recordsRejected, bytes per table). Each Airbyte attempt also appears as a sub task run named attempt-<n>. A job that ends in any state other than SUCCEEDED fails the task, but its outputs are kept, so errors and finally branches and allowFailure: true can still read them. Set onPartialSuccess: WARNING to end in WARNING instead when the job failed after committing records. Failure messages keep Airbyte's externalMessage (truncated to 2 KiB) and never include its internalMessage or stack trace. finalJobStatus is deprecated in favor of job.status. New metrics: duration, records.rejected and attempts.failed.

Airbyte Cloud — cloud.jobs.Sync triggers a Cloud sync by connectionId and waits by default. cloud.jobs.Reset resets a connection's state. Both support wait, maxDuration, and pollFrequency with the same defaults. Both also return connection, connectionSnapshot and jobUrl (a link to the job in Airbyte Cloud), read through the Airbyte Cloud API without ever reading connector configurations. cloud.jobs.Sync emits table assets for the replicated streams when the job succeeds (Enterprise Edition); cloud.jobs.Reset emits none. Airbyte Cloud's public API has no per-stream or per-attempt statistics, so there is no attempts output, no per-attempt sub task run and no onPartialSuccess. On Airbyte Cloud, the SDK only reads sources and destinations of connectors it knows: if one cannot be read, the task logs a warning and still returns the connection, jobUrl and assets, with that side's connector type left null. **Breaking change: ** both tasks now fail when the Airbyte job ends FAILED or CANCELLED, keeping their outputs; use allowFailure: true or an errors branch to keep the previous behavior.

Connection snapshot, job link and assets (self-hosted) — connections.Sync and connections.CheckStatus read the connection, its source and its destination through Airbyte's config API (/api/v1/connections/get, /api/v1/sources/get, /api/v1/destinations/get) and return connection (a short summary), connectionSnapshot (an internal storage JSON file with the replicated streams and their sync modes) and jobUrl (a link to the job logs in the Airbyte web app, built from webappUrl, which defaults to url). Connector configurations (hosts, passwords, keys) are never read, stored or returned. The snapshot is best-effort: if the credentials cannot read these endpoints, the task logs a warning, leaves these outputs null and carries on. connections.CheckStatus resolves the connection from the job's configId. When Kestra runs with assets enabled (Enterprise Edition), every replicated stream that committed records is emitted as a io.kestra.plugin.ee.assets.Table asset, with the source stream as its input; the asset id is <destinationId>.<namespace>.<table> and uses Airbyte's logical names, so it can differ from the physical table name when the destination normalizes names, and it does not match the ids other plugins emit for the same table. Assets are emitted only when the task ends in SUCCESS or WARNING: Kestra drops the assets of a FAILED task run, so none are emitted for a failed job. Their row and byte counts are the same per-stream figures as metadata, which only count successful attempts, so a job that ended in WARNING (partial success, no successful attempt) emits no asset.