GCS to Dataproc to BigQuery Lakehouse Refresh

Run the full GCP lakehouse chain, raw GCS data through a Dataproc Serverless PySpark transform into a curated BigQuery gold table, as a single orchestrated Kestra flow. This blueprint solves the classic medallion problem of stitching together object storage, a Spark transform, and a warehouse load that each have their own tooling and no shared scheduler, lineage, or failure handling. Instead of cron jobs and glue scripts, you get one declarative YAML pipeline that enforces the data dependency, retries on transient cloud errors, and notifies your team on every run.

How it works

1. The transform task (io.kestra.plugin.gcp.dataproc.batches.PySparkSubmit) submits a serverless PySpark batch to Dataproc in your configured region, running gs://<bucket>/jobs/transform.py. It reads raw files from the raw/ prefix and writes curated Parquet to the curated/ prefix in the same GCS bucket.
2. The load_gold task (io.kestra.plugin.gcp.bigquery.LoadFromGcs) loads the curated *.parquet files into the analytics.events_gold table using format: PARQUET and writeDisposition: WRITE_TRUNCATE, so the gold table always reflects the latest run.
3. The notify task (io.kestra.plugin.microsoft365.teams.TeamsIncomingWebhook) posts a Microsoft Teams message confirming the refresh, while an errors handler posts a failure alert if any step breaks.

Shared GCP credentials are set once via pluginDefaults for io.kestra.plugin.gcp (projectId and serviceAccount). A Schedule trigger (cron 0 4 * * *, shipped disabled) drives the daily refresh.

What you get

• A reproducible raw to curated to gold pipeline across GCS, Dataproc, and BigQuery.
• Serverless Spark with no cluster to provision, scale, or tear down.
• Idempotent gold loads via WRITE_TRUNCATE.
• Built-in Teams notifications on success and failure.
• One execution history covering the whole chain.

Who it's for

• Data engineers building or maintaining a GCP medallion lakehouse.
• Analytics engineers who need a dependable refresh feeding BigQuery models.
• Platform teams standardizing serverless Spark plus warehouse loads.

Why orchestrate this with Kestra

Dataproc, GCS, and BigQuery each have their own controls, but none coordinates the others. Kestra ties them into one declarative flow: the load only runs after the transform succeeds, every task carries retries for transient cloud failures, and the full chain shares a single execution and lineage view. You can swap the time-based Schedule for an event trigger so the refresh fires the moment new data lands, something the Dataproc batch scheduler alone cannot do.

Prerequisites

A GCP project with Dataproc Serverless, a GCS bucket, and a BigQuery analytics dataset. A PySpark job at jobs/transform.py and raw inputs under raw/ in the bucket.

Secrets

• GCP_PROJECT_ID: GCP project hosting Dataproc, GCS, and BigQuery.
• GCP_SERVICE_ACCOUNT: service account key JSON with Dataproc, GCS, and BigQuery access.
• GCP_REGION: region for the serverless batch.
• GCS_BUCKET: bucket holding the raw/ and curated/ prefixes plus the PySpark job.
• TEAMS_WEBHOOK_URL: Microsoft Teams incoming webhook URL.

Quick start

1. Set the secrets above in your Kestra namespace.
2. Point mainPythonFileUri, the args input and output prefixes, and destinationTable at your data.
3. Enable the Schedule trigger (or replace it with a GCS object-arrival trigger) once configured.

How to extend

• Replace the Schedule with a GCS trigger on a _SUCCESS marker to make the refresh event-driven.
• Use BigQuery partition decorators for incremental, partition-scoped loads instead of full truncation.
• Add a data-quality check task between transform and load to gate bad batches.
• Fan out multiple PySpark batches for a multi-source bronze-to-silver layer.

Links

• Google Cloud Platform plugin docs
• Dataproc orchestration with Kestra