Airbyte Sync

Airbyte Sync
Certified
Run an Airbyte connection sync

Starts a sync for an Airbyte connection and, by default, waits for the job to finish. Polling runs every second for up to 60 minutes unless you change pollFrequency or maxDuration. If a sync is already running for the connection, the task adopts it and polls it to completion instead of failing or blindly re-triggering — see onActiveSync

yaml
type: io.kestra.plugin.airbyte.connections.Sync

Examples

Trigger an Airbyte sync and wait for completion

yaml
id: airbyte_sync
namespace: company.team

tasks:
  - id: sync
    type: io.kestra.plugin.airbyte.connections.Sync
    url: http://localhost:8080
    connectionId: e3b1ce92-547c-436f-b1e8-23b6936c12cd

Trigger a single Airbyte sync on schedule

yaml
id: airbyte_sync
namespace: company.team

tasks:
  - id: data_ingestion
    type: io.kestra.plugin.airbyte.connections.Sync
    connectionId: e3b1ce92-547c-436f-b1e8-23b6936c12ab
    url: http://host.docker.internal:8000/
    username: "{{ secret('AIRBYTE_USERNAME') }}"
    password: "{{ secret('AIRBYTE_PASSWORD') }}"

triggers:
  - id: every_minute
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "*/1 * * * *"

Fail immediately instead of adopting an already running sync

yaml
id: airbyte_sync_fail_on_active
namespace: company.team

tasks:
  - id: sync
    type: io.kestra.plugin.airbyte.connections.Sync
    url: http://localhost:8080
    connectionId: e3b1ce92-547c-436f-b1e8-23b6936c12cd
    onActiveSync: FAIL

Properties

connectionId *string

Connection ID

Airbyte connection ID to sync

url *string

Airbyte API URL

Base URL of the Airbyte instance to call. This value is rendered from the current run context

applicationCredentials

Application credentials

Client credentials used to request an Airbyte application access token from /api/v1/applications/token. Use this instead of a static token when Airbyte application authentication is enabled

Definitions

io.kestra.plugin.airbyte.AbstractAirbyteConnection-ApplicationCredentials

clientId*string

Client ID

Airbyte application client ID

clientSecret*string

Client secret

Airbyte application client secret. Store this value in a Secret

maxDuration string

DefaultPT1H

Maximum wait duration

Maximum total time to wait when wait is enabled. Defaults to 60 minutes. When a job is adopted from an already running sync, this duration is counted from the moment it is adopted, not from the job's actual start time

onActiveSync string

DefaultADOPT

Possible Values

ADOPTFAILSKIP

Behavior when a sync is already running

Controls what happens when a sync is already running for this connection:

ADOPT (default): attach to the in-flight job and poll it to completion instead of triggering a new sync.
FAIL: fail the task immediately.
SKIP: succeed immediately without waiting, with alreadyRunning set to true and no jobId.

options

HTTP client options

HTTP client configuration applied to Airbyte API requests

Definitions

io.kestra.core.http.client.configurations.HttpConfiguration

allowFailedbooleanstring

Defaultfalse

If true, allow a failed response code (response code >= 400)

allowedResponseCodesarray

SubTypeinteger

List of response code allowed for this request

auth

The authentication to use.

io.kestra.core.http.client.configurations.BasicAuthConfiguration

type*object

passwordstring

The password for HTTP basic authentication.

usernamestring

The username for HTTP basic authentication.

io.kestra.core.http.client.configurations.BearerAuthConfiguration

type*object

tokenstring

The token for bearer token authentication.

io.kestra.core.http.client.configurations.DigestAuthConfiguration

type*object

passwordstring

The password for HTTP Digest authentication.

usernamestring

The username for HTTP Digest authentication.

defaultCharsetstring

DefaultUTF-8

The default charset for the request.

java.nio.charset.Charset

enabledTcpExtendedKeepAlivebooleanstring

Defaulttrue

Whether to enable TCP Keep-Alive extended socket options (TCP_KEEPIDLE, TCP_KEEPINTERVAL, TCP_KEEPCOUNT).

Set to false when running on Windows workers, as these extended socket options are not supported by the Windows JDK and will cause connection failures.

followRedirectsbooleanstring

Defaulttrue

Whether redirects should be followed automatically.

logsarray

SubTypestring

Possible Values

REQUEST_HEADERSREQUEST_BODYRESPONSE_HEADERSRESPONSE_BODY

The enabled log.

proxy

The proxy configuration.

io.kestra.core.http.client.configurations.ProxyConfiguration

addressstring

The address of the proxy server.

passwordstring

The password for proxy authentication.

portintegerstring

The port of the proxy server.

typestring

DefaultDIRECT

Possible Values

DIRECTHTTPSOCKS

The type of proxy to use.

usernamestring

The username for proxy authentication.

ssl

The SSL request options

io.kestra.core.http.client.configurations.SslOptions

insecureTrustAllCertificatesbooleanstring

Whether to disable checking of the remote SSL certificate.

Only applies if no trust store is configured. Note: This makes the SSL connection insecure and should only be used for testing. If you are using a self-signed certificate, set up a trust store instead.

timeout

The timeout configuration.

io.kestra.core.http.client.configurations.TimeoutConfiguration

connectTimeoutstring

The time allowed to establish a connection to the server before failing.

readIdleTimeoutstring

DefaultPT5M

The time allowed for a read connection to remain idle before closing it.

password string

Basic auth password

Password for Airbyte basic authentication. Store this value in a Secret

pollFrequency string

DefaultPT1S

Poll frequency

Interval between status checks while waiting for the job to finish. Defaults to 1 second

token string

Bearer token

Bearer token used for Airbyte API requests. This value is rendered from the current run context

username string

Basic auth username

Username for Airbyte basic authentication. If both basic auth and a bearer token are configured, basic auth is sent last and takes precedence

wait booleanstring

Defaulttrue

Wait for completion

If true, wait for the Airbyte job to reach a terminal state before the task completes. Defaults to true

Outputs

adopted boolean

Adopted

Whether jobId refers to a sync that was already running and got adopted, rather than a sync newly triggered by this task

alreadyRunning boolean

Already running

Whether a sync was already running for the connection, either adopted (onActiveSync: ADOPT) or skipped (onActiveSync: SKIP)

jobId integer

Job ID

Airbyte job ID for the sync. Set for every outcome except onActiveSync: SKIP, whether the job was newly triggered or adopted from an already running sync

Metrics

attempts.count counter

Unitattempt

Number of attempts made during the Airbyte sync (emitted when wait is enabled)

bytes.emitted counter

Unitbyte

Number of bytes emitted during processing (emitted when wait is enabled)

records.committed counter

Unitrecord

Number of records successfully committed (emitted when wait is enabled)

records.emitted counter

Unitrecord

Number of records emitted during processing (emitted when wait is enabled)

state.emitted counter

Unitmessage

Number of state messages emitted (emitted when wait is enabled)

Scrapy CLI

Tasks that run Scrapy spiders and CLI commands to scrape data from websites.

Data

Supabase

Tasks that call Supabase APIs for data and storage operations.

Data

Airbyte SyncCertifiedRun an Airbyte connection sync

More Plugins in this Category

Scrapy CLI

Supabase

Airbyte Sync
Certified
Run an Airbyte connection sync