Run a Scheduled ETL Sync with Airbyte Cloud

Keep your data warehouse fresh by triggering an Airbyte Cloud connection sync on a recurring schedule. This blueprint runs a single Airbyte Cloud connection as a scheduled ETL job, so source data is ingested automatically without anyone clicking "Sync" in the Airbyte UI. It is a clean starting point for incremental data ingestion, operational analytics, and ELT pipelines where one source needs to refresh reliably and often.

How it works

1. The every_minute trigger (io.kestra.plugin.core.trigger.Schedule) fires on a cron expression (*/1 * * * * in this example) to kick off the flow on a fixed cadence.
2. The data_ingestion task (io.kestra.plugin.airbyte.cloud.jobs.Sync) calls the Airbyte Cloud API to start a sync for the connection identified by connectionId, authenticating with the token property.
3. The task waits for the Airbyte Cloud job to finish, so the Kestra execution reflects the real sync outcome (success or failure).

What you get

• Hands-off, scheduled ingestion for one Airbyte Cloud connection.
• A single execution per sync that you can search, replay, and audit.
• A foundation you can chain into downstream transformations or analytics.
• Near-real-time ingestion patterns by simply tightening the cron.

Who it's for

• Data engineers running Airbyte Cloud connections who want code-defined scheduling.
• Analytics engineers feeding dbt or warehouse models from synced sources.
• Platform teams centralizing ingestion orchestration across many tools.

Why orchestrate this with Kestra

Airbyte Cloud can schedule a connection on its own, but it cannot place that sync inside a larger pipeline. With Kestra you orchestrate the sync alongside everything that depends on it: add retries on transient API failures, fire event-driven triggers instead of fixed clocks, branch into transformation and quality-check tasks the moment ingestion succeeds, and capture full execution lineage and run history. The whole workflow is declarative YAML, versioned and reviewable, rather than settings buried in a vendor UI.

Prerequisites

• An Airbyte Cloud account with at least one configured connection.
• The connectionId of the connection you want to sync.

Secrets

• AIRBYTE_CLOUD_API_TOKEN: an Airbyte Cloud API key, used by the token property to authenticate the Sync task.

Quick start

1. Generate an Airbyte Cloud API key from the Airbyte Cloud developer portal.
2. Store it as the AIRBYTE_CLOUD_API_TOKEN secret in Kestra.
3. Replace the connectionId value with the connection you want to sync.
4. Adjust the cron on the every_minute trigger to your desired cadence.
5. Save and enable the flow, then watch the first scheduled execution run.

How to extend

• Add a downstream dbt or SQL task that runs only after the sync succeeds.
• Swap the Schedule trigger for a flow or webhook trigger to sync on demand.
• Add retry and failure alerting (Slack, email) for resilient ingestion.
• Fan out to multiple connections by adding more Sync tasks or a loop.

Links

• Airbyte plugin documentation
• Airbyte orchestration with Kestra