Airbyte moves the data. Kestra decides when each sync runs, what happens after it lands, and what to do when it fails. Trigger on any event, wait for terminal status, chain straight into dbt and reverse-ETL, and trace every connector run in one execution history.
Airbyte's built-in scheduler fires connector syncs on a cron cadence. Kestra runs above it: it triggers Sync the moment an upstream event lands, waits for the job to reach a terminal state, then chains straight into dbt transforms and Hightouch activation in one flow. A failed transform retries without re-syncing the connector, and every sync, build, and activation shares one execution ID and one place to debug.
Airbyte owns the connectors and the EL. Kestra owns the steps around the sync: when it triggers, what runs after it, who approves the load, and where the cross-tool audit trail lives.
Airbyte's scheduler fires on a fixed cadence whether the source is ready or not. Kestra triggers io.kestra.plugin.airbyte.connections.Sync the moment an upstream event lands, a webhook, an S3 file, a Slack command, or a database change. Ingestion starts when there is actually something to ingest.
Firing a sync and hoping is not orchestration. Sync waits for the job to reach a terminal state by default, polling every second; CheckStatus can poll a job started elsewhere. The next task only runs after the sync confirms success, so transforms never read half-loaded tables.
Airbyte's run history stops at the connector. It has no record of the dbt build that shaped the data or the Hightouch sync that activated it. Kestra runs ingestion, transform, and activation as steps in one flow with run IDs flowing forward and one shared execution ID.
A warehouse refresh often means a dozen connector syncs. Kestra fans them out with ForEach and concurrencyLimit, and allowFailure isolates a single failing connector so the rest still complete. A targeted alert routes to the owner of the broken connection.
A schema change forces a connection reset and a full re-sync. The cloud.jobs.Reset task clears connector state and cloud.jobs.Sync reloads, driven from a typed flow input so a backfill is a form submission, not a manual console click. The same flow handles the nightly incremental and the on-demand full reload.
When data looks stale, you need to know whether the sync ran, how many records moved, and what happened next. Airbyte tracks the connector job; Kestra surfaces its emitted metrics next to the dbt build and the activation sync, every step timestamped and attributed to a trigger source.
Patterns data engineering teams run in production today. Each one shows the flow end to end, with the real plugin classes in play.
Kestra runs the Airbyte sync, waits for terminal status, builds dbt models on the freshly landed data, then fires the Hightouch sync. Slack confirms. A dbt retry never re-runs the Airbyte sync.
Sync blocks until terminal status, so dbt never reads a half-loaded table.
A dbt retry never re-runs the sync; an activation retry never re-runs dbt.
Sync metrics, dbt results, and the activation run in one view.
Replace the cron schedule with a push. A webhook (or S3 arrival, or Slack command) fires the flow, Kestra runs the matching Airbyte connection, waits for completion, and branches on the result. The sync runs when the source is confirmed ready.
Ingestion fires on the event that matters, not the next interval.
The connection ID can come from trigger.body at run time.
An If task routes success and failure to different paths.
A warehouse refresh means many connections at once. Kestra fans them out via ForEach with a concurrencyLimit, runs each Sync, and isolates failures with allowFailure so one broken connector does not block the rest. The owner of the failing connection gets paged.
concurrencyLimit caps how many syncs run at once.
allowFailure keeps one bad connection from halting the batch.
Slack routes by connection so the right team is paged.
A schema change needs a connection reset and full reload. Kestra runs cloud.jobs.Reset to clear state, then cloud.jobs.Sync to reload, with the connection ID and full-vs-incremental mode as typed inputs. The nightly incremental and the on-demand backfill share one flow.
Reset clears state before Sync reloads.
Full-vs-incremental is a flow input, not a code change.
Trigger a backfill from a form without touching YAML.
Kestra runs the sync, waits, then checks row counts or runs a dbt source freshness test. If the load is short or stale, it halts and pages PagerDuty before any transform touches the data. Clean loads continue to the build.
A short or stale load halts the flow before dbt runs.
A dbt source freshness test decides pass or halt.
Stale ingestion pages the on-call, not just a log line.
Kestra has been a keystone to design complex execution flows while enhancing our Infrastructure as Code best practices. It now empowers our ingestion with Airbyte, transformation with dbt, and our Hightouch reverse-ETL jobs seamlessly.
| Capability | | Airbyte Scheduler | | |
|---|---|---|---|---|
| Trigger Airbyte on an upstream event | Native event triggers | Cron schedule only | Sensors, Python required | Event polling code |
| Wait for terminal sync status | Sync waits by default | Fire and forget | Custom op logic | Custom polling code |
| Chain sync with dbt and Hightouch | Native plugins | Sync only, no downstream | Asset-based, Python | Python code required |
| Parallel sync fan-out with isolation | ForEach + allowFailure | Per-connection schedules | Dynamic ops | Task mapping |
| Reset and backfill from an input | Reset + Sync tasks | Manual in the UI | Custom op | Python code required |
| Self-service backfills for analysts | Kestra Apps | UI for engineers | No native form layer | No native form layer |
| Cross-stack execution history | One view across tools | Airbyte-scoped | Dagster-scoped | Prefect-scoped |
Find answers to your questions right here, and don't hesitate to Contact Us if you couldn't find what you're looking for.
Integrate Airbyte using Kestra
When to choose Airbyte, Fivetran, or Kestra for data ingestion
Why data integration will never be fully solved, and what Fivetran, Airbyte, Singer, dlt and CloudQuery do about it
Announcing Kestra Technology Partnership with Airbyte
Trigger on any event, wait for terminal status, chain straight into dbt and Hightouch, and trace every connector run in one place. Open source, self-hosted, event-driven.
