Load GCS Files into BigQuery on Arrival

Turn a Google Cloud Storage bucket into an event source that feeds BigQuery automatically. The moment new files land in a landing prefix, this flow loads each one into a raw BigQuery table, moves the processed object to an archive prefix so it can never re-trigger, and confirms the batch in Slack. It removes the gap between data arriving in object storage and data being queryable in your warehouse, with no manual loads, no cron guesswork, and no duplicate ingestion.

How it works

1. The io.kestra.plugin.gcp.gcs.Trigger polls gs://{bucket}/landing/ every PT1M and fires on CREATE events, exposing the new objects as trigger.blobs. Its action: MOVE relocates each processed object to gs://{bucket}/archive/ so the same file is never detected twice.
2. The load_new_files task (io.kestra.plugin.core.flow.ForEach) iterates over trigger.blobs.
3. For each blob, io.kestra.plugin.gcp.bigquery.Load appends the file to analytics.landed_files using format: CSV, autodetect: true, createDisposition: CREATE_IF_NEEDED, and writeDisposition: WRITE_APPEND, so the table is created on first run and grown on every subsequent run.
4. The notify task (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) posts how many files were loaded.
5. An alert_on_failure error handler posts to Slack if any load fails.

What you get

• Event-driven ingestion: files become queryable rows seconds after they land.
• Exactly-once handling via archive MOVE, so reprocessing and duplicates are avoided.
• Hands-off schema management through BigQuery autodetect and CREATE_IF_NEEDED.
• Real-time success and failure visibility in Slack.

Who it's for

• Data engineers building landing-zone or ELT ingestion pipelines on Google Cloud.
• Analytics teams who need fresh raw tables without waiting on batch windows.
• Platform teams replacing brittle Cloud Functions or ad hoc scripts with declarative orchestration.

Why orchestrate this with Kestra

GCS object notifications and BigQuery scheduled queries can each move data, but neither gives you a single declarative pipeline that reacts to file arrival, fans out per object, loads, archives, alerts, and retries as one auditable unit. Kestra adds the event Trigger, ForEach fan-out, automatic retries, full execution lineage over every blob, and version-controlled YAML, filling the gap that a bucket watcher or a warehouse scheduler alone cannot cover.

Prerequisites

A GCS bucket with a landing/ prefix and a BigQuery analytics dataset. The destination table analytics.landed_files is created on the first load.

Secrets

• GCP_PROJECT_ID: GCP project hosting the bucket and dataset.
• GCP_SERVICE_ACCOUNT: service account key JSON with GCS read/write and BigQuery job access.
• GCS_BUCKET: bucket holding the landing and archive prefixes.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Set the four secrets above in your Kestra namespace.
2. Point the trigger from and moveDirectory at your bucket prefixes.
3. Adjust format (CSV, JSON, AVRO, PARQUET) and destinationTable to match your files.
4. Set disabled: false on the on_new_object trigger and drop a file into the landing prefix.

How to extend

• Switch action: MOVE to DELETE to remove processed files, or NONE with a stateKey to leave them in place.
• Lower the trigger interval for fresher detection, or set maxFiles to cap objects per run.
• Chain a dbt or BigQuery SQL task after the load to parse and model the raw landing table downstream.
• Add a routing condition on file extension to load different formats into different tables.

Links

• Plugin docs: https://kestra.io/plugins/plugin-gcp
• Orchestration: https://kestra.io/orchestration/gcs