Fivetran Connectors Sync

Fivetran Connectors Sync

Certified

Trigger and optionally watch connector sync

Starts a Fivetran connector sync through the Fivetran API. Can force-cancel and restart an in-progress sync, or re-attach to one already in progress with reattach. Waits for completion by default (up to 60 minutes) and fails if the connector reports a failure. Fails right away on a paused connector, since Fivetran accepts the sync request but starts nothing. A completion Fivetran reports while the connector is still syncing waits up to 1 minute for the syncing state to end before it is accepted, so the task does not return before the sync has finished, and a busy connector whose next sync starts right away still returns within that minute. With assets.enableAuto set and wait: true, emits one asset per table the connector writes, with the id database.schema.name so it joins the dbt model reading that table in the lineage graph, and the source table as its input: a Table asset for database sources (e.g. PostgreSQL, MySQL, SQL Server, Snowflake, BigQuery) or an External asset connectorId.schema.table otherwise. With wait: false no asset is emitted, since the sync has not finished when the task returns. If building or emitting an asset fails, assets.assetFailureBehavior applies: FAIL fails the task, WARN (the default) ends it in WARNING, IGNORE only logs at debug level. Whatever assets.enableAuto is set to, reads the connector's schema config once (one extra GET) to output a per-table breakdown file (tables, tableCount, enabledTableCount) and a freshness summary with the connector's alert codes; a failed schema read logs a warning and never fails the task.

yaml
type: io.kestra.plugin.fivetran.connectors.Sync
yaml
id: fivetran_sync
namespace: company.team

tasks:
  - id: sync
    type: io.kestra.plugin.fivetran.connectors.Sync
    apiKey: "{{ secret('FIVETRAN_API_KEY') }}"
    apiSecret: "{{ secret('FIVETRAN_API_SECRET') }}"
    connectorId: "connector_id"

Sync a connector and emit one lineage asset per table it writes.

yaml
id: fivetran_sync_with_assets
namespace: company.team

tasks:
  - id: sync
    type: io.kestra.plugin.fivetran.connectors.Sync
    apiKey: "{{ secret('FIVETRAN_API_KEY') }}"
    apiSecret: "{{ secret('FIVETRAN_API_SECRET') }}"
    connectorId: "connector_id"
    assets:
      enableAuto: true

Sync a connector, then read its freshness and per-table breakdown.

yaml
id: fivetran_sync_with_breakdown
namespace: company.team

tasks:
  - id: sync
    type: io.kestra.plugin.fivetran.connectors.Sync
    apiKey: "{{ secret('FIVETRAN_API_KEY') }}"
    apiSecret: "{{ secret('FIVETRAN_API_SECRET') }}"
    connectorId: "connector_id"
    wait: true

  - id: report
    type: io.kestra.plugin.core.log.Log
    message: |
      Alerts: {{ outputs.sync.freshness.alertCount }},
      enabled tables: {{ outputs.sync.enabledTableCount }} of {{ outputs.sync.tableCount }},
      breakdown file: {{ outputs.sync.tables }}

Re-attach to a sync already running on the connector instead of triggering a duplicate.

yaml
id: fivetran_sync_reattach
namespace: company.team

tasks:
  - id: sync
    type: io.kestra.plugin.fivetran.connectors.Sync
    apiKey: "{{ secret('FIVETRAN_API_KEY') }}"
    apiSecret: "{{ secret('FIVETRAN_API_SECRET') }}"
    connectorId: "connector_id"
    reattach: true
    reattachMaxAge: PT30M
Installation

Our standard Docker images already include this plugin. If you use a -slim image or another setup, install it with the following command: 

kestractl plugins get io.kestra.plugin:plugin-fivetran --compatible-for $KESTRA_VERSION

For Kestra CLI, replace LATEST with a specific version or version range to pin the plugin version. For kestractl, use your Kestra version with --compatible-for, or specify an exact plugin version.

Properties

Fivetran API key

Required; paired with apiSecret for HTTP Basic authentication.

Fivetran API secret

Required secret token used with apiKey for Basic authentication.

Connector ID

Identifier of the Fivetran connector to sync.

Assets this task consumes as inputs or produces as outputs, for lineage tracking and the asset graph (Enterprise Edition). A flow declaring this property on a task is rejected in the open-source edition.

Definitions
assetFailureBehaviorstring
Possible Values
IGNOREFAILWARN

Asset failure behavior

Behavior applied to the task state when a declared asset fails to render, emit, or be persisted (e.g. a lock conflict): FAIL escalates it to FAILED, WARN (default) warns it if it would otherwise succeed, IGNORE leaves the state untouched.

enableAutobooleanstring

Whether to auto-register assets referenced dynamically at runtime that are not statically declared in inputs or outputs.

inputsarray

The assets consumed as inputs.

id*string
Min length1
typestring
outputs

The assets produced as outputs.

id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*object
descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
id*string
Min length1
Max length150
type*string
Min length1

Custom asset type

descriptionstring
displayNamestring
metadataobject
Default{}
namespacestring
Min length1
Max length150
Defaulthttps://api.fivetran.com

Fivetran API base URL

Base endpoint for all requests. Defaults to https://api.fivetran.com; override for regional or private deployments.

Defaultfalse

Force restart if already syncing

When true, cancels a running sync before starting a new one. Default is false to skip if a sync is already running.

DefaultPT1S

Initial delay before the first retry

Backoff grows from this delay, capped at 30 seconds. Default: 1 second.

Default3

Maximum number of attempts on transient errors

Total attempts including the first call, not just retries. Default: 3.

DefaultPT1H

Maximum wait duration

Upper bound for waiting when wait is true. Default is 60 minutes.

HTTP client options

Optional Kestra HTTP configuration (timeouts, proxy) applied to Fivetran calls. Retries are configured separately via maxAttempts and initialRetryDelay.

Definitions
allowFailedbooleanstring
Defaultfalse

If true, allow a failed response code (response code >= 400)

allowedResponseCodesarray
SubTypeinteger

List of response code allowed for this request

auth

The authentication to use.

type*object
passwordstring

The password for HTTP basic authentication.

usernamestring

The username for HTTP basic authentication.

type*object
tokenstring

The token for bearer token authentication.

type*object
passwordstring

The password for HTTP Digest authentication.

usernamestring

The username for HTTP Digest authentication.

defaultCharsetstring
DefaultUTF-8

The default charset for the request.

enabledTcpExtendedKeepAlivebooleanstring
Defaulttrue

Whether to enable TCP Keep-Alive extended socket options (TCP_KEEPIDLE, TCP_KEEPINTERVAL, TCP_KEEPCOUNT).

Set to false when running on Windows workers, as these extended socket options are not supported by the Windows JDK and will cause connection failures.

followRedirectsbooleanstring
Defaulttrue

Whether redirects should be followed automatically.

logsarray
SubTypestring
Possible Values
REQUEST_HEADERSREQUEST_BODYRESPONSE_HEADERSRESPONSE_BODY

The enabled log.

proxy

The proxy configuration.

addressstring

The address of the proxy server.

passwordstring

The password for proxy authentication.

portintegerstring

The port of the proxy server.

typestring
DefaultDIRECT
Possible Values
DIRECTHTTPSOCKS

The type of proxy to use.

usernamestring

The username for proxy authentication.

retry

Retry strategy for HTTP requests.

retry is disabled by default.

Retry with a fixed delay between attempts.

interval*string
Formatduration
type*object
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse

Retry with exponentially increasing delays between attempts.

interval*string
Formatduration
maxInterval*string
Formatduration
type*object
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
delayFactornumber
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse

Retry with a random delay within a configurable range between attempts.

maxInterval*string
Formatduration
minInterval*string
Formatduration
type*object
behaviorstring
DefaultRETRY_FAILED_TASK
Possible Values
RETRY_FAILED_TASKCREATE_NEW_EXECUTION
maxAttemptsinteger
Minimum>= 1
maxDurationstring
Formatduration
warningOnRetryboolean
Defaultfalse
retryOnStatusCodesarray
SubTypeinteger
Default[502,503,504]

HTTP status codes that should be retried.

Used for idempotent methods (GET, HEAD) without their own entry in retryOnStatusCodesByMethod. Non-idempotent methods are never retried on a status code unless listed in retryOnStatusCodesByMethod. Defaults to 502, 503 and 504 for idempotent methods (GET, HEAD) only.

retryOnStatusCodesByMethodobject
SubTypearray

Per-HTTP-method overrides for which status codes should be retried.

Keys are HTTP methods (e.g. GET, POST). A method with no entry here falls back to retryOnStatusCodes. This lets a caller retry more broadly for safe, read-only methods (for example, every 5xx and a 429 for GET/HEAD) while keeping a narrower list for methods that write, where retrying a request that may have already been processed by the origin risks duplicating the effect. Example: {GET: [429, 500, 502, 503, 504], HEAD: [429, 500, 502, 503, 504]}.

retryableTransportFailureMethodsarray
SubTypestring
Default["GET","HEAD"]

HTTP methods for which any transport-level failure may be retried.

A transport-level failure is a connection or timeout error rather than an HTTP response (e.g. a read timeout or a connection reset mid-request). For a method listed here, any such failure is retried. Defaults to GET and HEAD HTTP methods.

ssl

The SSL request options

insecureTrustAllCertificatesbooleanstring

Whether to disable checking of the remote SSL certificate.

Only applies if no trust store is configured. Note: This makes the SSL connection insecure and should only be used for testing. If you are using a self-signed certificate, set up a trust store instead.

timeout

The timeout configuration.

connectTimeoutstring

The time allowed to establish a connection to the server before failing.

readIdleTimeoutstring
DefaultPT5M

The time allowed for a read connection to remain idle before closing it.

DefaultPT5S

Poll frequency

Interval between connector status checks while waiting for the sync to complete. Default is 5 seconds.

Defaultfalse

Reattach to an already in-progress sync instead of starting a new one

When true, adopts any sync already running on the connector -- including one Fivetran started itself on its own schedule, not just a sync a previous run of this task triggered -- instead of triggering a new one. Fivetran exposes no way to tell one sync run apart from another, so adoption matches on the connector's own sync_state: syncing as reported by Fivetran, not on this execution's identity. This covers a worker-loss resubmit, a retry, a replay, a manual re-run, or two executions driving the same connector: whichever one observes the connector already syncing waits on it instead of triggering a duplicate. Default is false. force: true always takes precedence over reattach and starts a new sync regardless.

Maximum age of an in-progress sync that can still be re-attached to

Used only when reattach is true. Fivetran reports no sync start time, so the connector's last completion timestamp is used as a conservative upper bound on how long the current sync has been running: when that bound exceeds reattachMaxAge, the in-progress sync is treated as stale: it is cancelled and a new sync is started (the trigger is sent with force: true) instead of adopting it. A connector that has never completed a sync has no such upper bound to check, so once reattachMaxAge is set it is always treated as stale and never adopted -- also taking the forced, cancel-and-restart path. Default is unbounded: any in-progress sync is adopted, however long it has been running.

Defaulttrue

Wait for sync completion

When true (default), poll the connector until the sync finishes to capture status and logs. Set to false to return once the sync has been triggered, without waiting for it. succeededAt is then null, since the sync is still running on Fivetran. No assets are emitted in that case, since the sync has not finished. To record lineage after the sync, add a downstream io.kestra.plugin.fivetran.connectors.Status task with assets.enableAuto.

Connector ID

The connector this task synced.

Number of enabled tables

Tables that are enabled and belong to an enabled schema, i.e. the ones written to the destination. Null when the schema config could not be read.

Connector freshness

State of the connector when the task returned. With wait: true it is read after the sync completed. With wait: false it is the snapshot taken before the sync was triggered, so fresh is null and freshness.succeededAt is the connector's last known success, which predates the triggered sync. The top-level succeededAt is the completion of the sync this task observed.

Definitions
alertCodesarray
SubTypestring

Alert codes

De-duplicated codes of the connector's warnings and tasks. Alert messages are left out to keep the output bounded.

alertCountinteger
Default0

Number of alerts

Size of the connector's status.warnings plus status.tasks.

failedAtstring
Formatdate-time

Timestamp of the last failed sync

freshboolean

Whether the connector's data is fresh

True when wait is true, since a sync that just succeeded is fresh by definition. Null when wait is false, because the sync is still running.

pausedboolean

Whether the connector is paused

setupStatestring

Current setup state

As reported by Fivetran, e.g. connected, broken, incomplete.

succeededAtstring
Formatdate-time

Timestamp of the last successful sync

The connector's last known success, taken from the connector snapshot. With wait: false it predates the sync this task triggered; use the top-level succeededAt for the sync the task itself observed.

syncFrequencyinteger

Sync frequency in minutes

syncStatestring

Current sync state

As reported by Fivetran, e.g. scheduled, syncing, paused, rescheduled.

updateStatestring

Current update state

As reported by Fivetran, e.g. on_schedule, delayed.

Defaultfalse

Whether an already in-progress sync was re-attached to

True when reattach adopted a sync already running on the connector instead of triggering a new one.

Formatdate-time

Timestamp of the sync this task waited for

The completion of the sync this task observed, as reported by Fivetran. Null when wait is false, since the sync is still running on Fivetran when the task returns. See freshness.succeededAt for the connector's last known success.

Number of tables

Every table the connector reports, enabled or not. Null when the schema config could not be read.

Formaturi

Per-table breakdown file

URI of an Ion file in internal storage with one row per table of the connector: schema, schemaInDestination, schemaEnabled, table, tableInDestination, enabled, syncMode and enabledPatchAllowed. Null when the schema config could not be read.

Whether the task re-attached to an already in-progress sync (1) instead of triggering a new one (0).

More Plugins in this Category

Firecrawl
Firecrawl

Tasks that scrape and crawl websites into clean, LLM-ready content with Firecrawl.

Data

2 tasks

Liquibase
Liquibase

Tasks that run Liquibase change management commands.

Data

2 tasks

8 blueprints