Airbyte Sync

Airbyte Sync

Certified

Run an Airbyte connection sync

Starts a sync for an Airbyte connection and, by default, waits for the job to finish. Polling runs every second for up to 60 minutes unless you change pollFrequency or maxDuration

yaml
type: io.kestra.plugin.airbyte.connections.Sync
yaml
id: airbyte_sync
namespace: company.team

tasks:
  - id: sync
    type: io.kestra.plugin.airbyte.connections.Sync
    url: http://localhost:8080
    connectionId: e3b1ce92-547c-436f-b1e8-23b6936c12cd

Trigger a single Airbyte sync on schedule

yaml
id: airbyte_sync
namespace: company.team

tasks:
  - id: data_ingestion
    type: io.kestra.plugin.airbyte.connections.Sync
    connectionId: e3b1ce92-547c-436f-b1e8-23b6936c12ab
    url: http://host.docker.internal:8000/
    username: "{{ secret('AIRBYTE_USERNAME') }}"
    password: "{{ secret('AIRBYTE_PASSWORD') }}"

triggers:
  - id: every_minute
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "*/1 * * * *"
Properties

Connection ID

Airbyte connection ID to sync

Airbyte API URL

Base URL of the Airbyte instance to call. This value is rendered from the current run context

Application credentials

Client credentials used to request an Airbyte application access token from /api/v1/applications/token. Use this instead of a static token when Airbyte application authentication is enabled

Definitions
clientId*Requiredstring

Client ID

Airbyte application client ID

clientSecret*Requiredstring

Client secret

Airbyte application client secret. Store this value in a Secret

Defaulttrue

Fail on active sync

If true, fail when Airbyte reports that a sync is already running for the connection. If false, the task succeeds with alreadyRunning set to true

DefaultPT1H

Maximum wait duration

Maximum total time to wait when wait is enabled. Defaults to 60 minutes

HTTP client options

HTTP client configuration applied to Airbyte API requests

Definitions
allowFailedbooleanstring
Defaultfalse

If true, allow a failed response code (response code >= 400)

allowedResponseCodesarray
SubTypeinteger

List of response code allowed for this request

auth

The authentication to use.

type*Requiredobject
passwordstring

The password for HTTP basic authentication.

usernamestring

The username for HTTP basic authentication.

type*Requiredobject
tokenstring

The token for bearer token authentication.

type*Requiredobject
passwordstring

The password for HTTP Digest authentication.

usernamestring

The username for HTTP Digest authentication.

basicAuthPasswordDeprecatedstring

The password for HTTP basic authentication. Deprecated, use auth property with a BasicAuthConfiguration instance instead.

basicAuthUserDeprecatedstring

The username for HTTP basic authentication. Deprecated, use auth property with a BasicAuthConfiguration instance instead.

connectTimeoutDeprecatedstring
Formatduration

The time allowed to establish a connection to the server before failing.

connectionPoolIdleTimeoutDeprecatedstring
Formatduration

The time an idle connection can remain in the client's connection pool before being closed.

defaultCharsetstring
DefaultUTF-8

The default charset for the request.

enabledTcpExtendedKeepAlivebooleanstring
Defaulttrue

Whether to enable TCP Keep-Alive extended socket options (TCP_KEEPIDLE, TCP_KEEPINTERVAL, TCP_KEEPCOUNT).

Set to false when running on Windows workers, as these extended socket options are not supported by the Windows JDK and will cause connection failures.

followRedirectsbooleanstring
Defaulttrue

Whether redirects should be followed automatically.

logLevelDeprecatedstring
Possible Values
ALLTRACEDEBUGINFOWARNERROROFFNOT_SPECIFIED

The log level for the HTTP client.

logsarray
SubTypestring
Possible Values
REQUEST_HEADERSREQUEST_BODYRESPONSE_HEADERSRESPONSE_BODY

The enabled log.

maxContentLengthDeprecatedinteger

The maximum content length of the response.

proxy

The proxy configuration.

addressstring

The address of the proxy server.

passwordstring

The password for proxy authentication.

portintegerstring

The port of the proxy server.

typestring
DefaultDIRECT
Possible Values
DIRECTHTTPSOCKS

The type of proxy to use.

usernamestring

The username for proxy authentication.

proxyAddressDeprecatedstring

The address of the proxy server.

proxyPasswordDeprecatedstring

The password for proxy authentication.

proxyPortDeprecatedinteger

The port of the proxy server.

proxyTypeDeprecatedstring
Possible Values
DIRECTHTTPSOCKS

The type of proxy to use.

proxyUsernameDeprecatedstring

The username for proxy authentication.

readIdleTimeoutDeprecatedstring
Formatduration

The time allowed for a read connection to remain idle before closing it.

readTimeoutDeprecatedstring
Formatduration

The maximum time allowed for reading data from the server before failing.

ssl

The SSL request options

insecureTrustAllCertificatesbooleanstring

Whether to disable checking of the remote SSL certificate.

Only applies if no trust store is configured. Note: This makes the SSL connection insecure and should only be used for testing. If you are using a self-signed certificate, set up a trust store instead.

timeout

The timeout configuration.

connectTimeoutstring

The time allowed to establish a connection to the server before failing.

readIdleTimeoutstring
DefaultPT5M

The time allowed for a read connection to remain idle before closing it.

Basic auth password

Password for Airbyte basic authentication. Store this value in a Secret

Reference (ref) of the pluginDefaults to apply to this task.

DefaultPT1S

Poll frequency

Interval between status checks while waiting for the job to finish. Defaults to 1 second

Bearer token

Bearer token used for Airbyte API requests. This value is rendered from the current run context

Basic auth username

Username for Airbyte basic authentication. If both basic auth and a bearer token are configured, basic auth is sent last and takes precedence

Defaulttrue

Wait for completion

If true, wait for the Airbyte job to reach a terminal state before the task completes. Defaults to true

Already running

Whether Airbyte reported that a sync was already running for the connection

Job ID

Airbyte job ID created by the sync request

More Plugins in this Category

Clickhouse
Clickhouse

Tasks that connect to ClickHouse via JDBC to run queries and loads.

Data

5 tasks

2 blueprints

Weaviate
Weaviate

Tasks that manage and query vectors in Weaviate.

DataAI

4 tasks

2 blueprints