Automate Cross-Cloud ETL from Azure Blob Storage to BigQuery with dbt Cloud

Move data across clouds without a fragile chain of schedulers. This blueprint watches an Azure Blob Storage container for new CSV files, loads each one into a Google BigQuery table the moment it lands, and then kicks off a dbt Cloud job to turn raw rows into analytics-ready models. It closes the common gap in Azure-to-GCP data pipelines: reacting to files as they arrive instead of polling on a fixed clock, and chaining ingestion to transformation as one observable run.

How it works

• A io.kestra.plugin.azure.storage.blob.Trigger polls the stage container under the marketplace/ prefix every PT30S. When new blobs appear, it starts an execution and exposes them as trigger.blobs.
• The trigger action is MOVE, so each processed blob is relocated to archive/marketplace/ in the same container, preventing duplicate ingestion and keeping the staging area clean.
• A io.kestra.plugin.core.flow.ForEach task iterates over the blob URIs (trigger.blobs | jq('.[].uri')) with concurrencyLimit: 0 for parallel processing.
• For each file, io.kestra.plugin.gcp.bigquery.Load loads the CSV into the destination table using autodetect: true for schema inference, writeDisposition: WRITE_APPEND, and CSV options (fieldDelimiter, encoding: UTF-8, allowJaggedRows).
• Once ingestion finishes, io.kestra.plugin.dbt.cloud.TriggerRun triggers a dbt Cloud job with wait: true, so the run blocks until dbt completes and surfaces model and test results.

What you get

• Hands-off ingestion the instant files land in Azure Blob Storage.
• Parallel, high-throughput CSV loads into BigQuery.
• Automatic archival of processed files to avoid reprocessing.
• A single execution that spans ingestion and dbt transformation, with full run lineage.

Who it's for

• Data engineers building cross-cloud ETL and ELT pipelines.
• Analytics teams running BigQuery plus dbt Cloud as their modern data stack.
• Platform teams migrating or syncing data from Azure to GCP.

Why orchestrate this with Kestra

The Azure Blob trigger reacts to events, not a cron clock, so data flows as soon as it arrives. Kestra adds retries on transient cloud failures, end-to-end lineage across Azure, BigQuery, and dbt in one execution, and declarative YAML you can version control. Neither the BigQuery scheduler nor dbt Cloud's own scheduler can watch an Azure container and chain a load into a transformation as a single observable pipeline. Kestra fills that gap.

Prerequisites

• An Azure Storage account with a container holding CSV files.
• A Google Cloud project with BigQuery enabled and a service account that can load tables.
• A dbt Cloud account with a job to trigger.

Secrets

• AZURE_CONNECTION_STRING: Azure Storage connection string for the blob trigger.
• GCP_CREDS: GCP service account JSON for the BigQuery load.
• DBT_CLOUD_ACCOUNT_ID: dbt Cloud account ID.
• DBT_CLOUD_API_TOKEN: dbt Cloud API token.

Quick start

1. Add the four secrets above to your Kestra instance.
2. Set the Azure endpoint, container, and prefix to match your storage.
3. Update the BigQuery destinationTable, projectId, and CSV options.
4. Replace the jobId with your dbt Cloud transformation job.
5. Upload a CSV to the watched prefix and confirm it loads, archives, and runs dbt.

How to extend

• Swap format: CSV and csvOptions for JSON, Avro, or Parquet ingestion.
• Add validation or quality-check tasks between the load and the dbt run.
• Notify Slack or email on completion or failure.
• Fan out to multiple destination tables or datasets by parameterizing the load.

Links

• Plugin docs: GCP
• BigQuery orchestration with Kestra