Run Parallel ETL Ingestion with Airbyte to Sync Multiple Data Sources

Run several Airbyte ELT syncs at the same time instead of one after another. This blueprint triggers multiple Airbyte connections in parallel from a single Kestra flow, so Salesforce, Google Analytics, and Facebook Ads data all land in your warehouse on the same schedule without serializing each load. It solves the common bottleneck where independent source syncs queue behind each other and stretch the ingestion window far longer than it needs to be.

How it works

• A data_ingestion task of type io.kestra.plugin.core.flow.Parallel wraps three child tasks and launches them concurrently.
• Each child task is an io.kestra.plugin.airbyte.connections.Sync task (salesforce, google_analytics, facebook_ads) that triggers an Airbyte sync by its connectionId.
• Each io.kestra.plugin.airbyte.connections.Sync task is configured with the Airbyte server url plus username and password pulled from secrets.
• Because the three syncs sit inside Parallel, they run together and the flow completes when the slowest one finishes.

What you get

• Concurrent ingestion of multiple Airbyte connections in one run.
• A shorter overall ingestion window versus sequential syncs.
• Consistent server config and credentials applied to every sync task.
• A single observable flow with per-sync status, logs, and retry behavior.

Who it's for

• Data engineers consolidating SaaS sources into a warehouse or lake.
• Analytics engineers who need fresh source data before transformations run.
• Platform teams standardizing how Airbyte syncs are triggered and monitored.

Why orchestrate this with Kestra

Airbyte's built-in scheduler runs each connection on its own independent timer, which cannot coordinate several connections as one unit, fan them out in parallel, or hand off cleanly to downstream transformation and quality steps. With Kestra you trigger all the syncs from one declarative YAML flow, run them in parallel, add retries on transient API failures, react to event triggers (flow, webhook, or schedule) rather than fixed per-connection timers, and keep full execution lineage across every sync in one place.

Prerequisites

• A running Airbyte server reachable from Kestra (the default url points to http://host.docker.internal:8000/).
• Three configured Airbyte connections and their connectionId values.
• The Airbyte plugin available in your Kestra instance.

Secrets

• AIRBYTE_USERNAME: username for the Airbyte server.
• AIRBYTE_PASSWORD: password for the Airbyte server.

Quick start

1. Store AIRBYTE_USERNAME and AIRBYTE_PASSWORD as secrets in Kestra.
2. Update the url on each Sync task to point at your Airbyte server.
3. Replace the three connectionId values with connection IDs from your Airbyte workspace.
4. Add the flow to your namespace and run it, then watch the three syncs execute in parallel.

How to extend

• Add or remove io.kestra.plugin.airbyte.connections.Sync tasks inside Parallel to cover more connections.
• Set the concurrent property on io.kestra.plugin.core.flow.Parallel to cap how many syncs run at once.
• Add a Schedule trigger to run ingestion on a cadence, or a flow trigger to chain it after upstream events.
• Append dbt or SQL transformation tasks after data_ingestion so models build as soon as the syncs complete.

Links

• Airbyte plugin docs
• Airbyte orchestration with Kestra