Run a Spark Batch on Dataproc Serverless

Run Apache Spark on Google Cloud Dataproc Serverless without provisioning, sizing, or tearing down a single cluster. This blueprint submits a Spark batch from a Kestra schedule: Dataproc Serverless provisions its own ephemeral compute per job, runs your Spark application, and releases the compute automatically, so cost tracks actual work rather than cluster uptime. Kestra wraps the batch with the scheduling, retries, lineage, and alerting that a raw API submission or the Dataproc console cannot give you, and Slack reports the final state of every run.

How it works

• The submit_spark task (io.kestra.plugin.gcp.dataproc.batches.SparkSubmit) submits a Spark batch to Dataproc Serverless in the configured region. The sample runs the bundled org.apache.spark.examples.SparkPi class from spark-examples.jar with an argument of 1000, and names the batch after the execution id.
• The notify task (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) posts the batch state from outputs.submit_spark.state to Slack so the result is visible without opening the cloud console.
• pluginDefaults apply projectId and serviceAccount to every GCP task, keeping credentials in one place.
• The nightly Schedule trigger (io.kestra.plugin.core.trigger.Schedule, cron 0 1 * * *, disabled by default) submits the batch on a cadence once you enable it.
• The alert_on_failure error handler posts a Slack message if the batch fails, so failures are never silent.

What you get

• Serverless Spark execution with zero cluster lifecycle to manage.
• Cost that tracks the job, not idle compute.
• Slack notifications on both success and failure.
• A reusable, declarative pattern you can point at any Spark application.

Who it's for

• Data engineers running batch Spark jobs on Google Cloud.
• Platform teams who want to avoid standing Dataproc clusters.
• Analytics teams orchestrating periodic transformations on GCP.

Why orchestrate this with Kestra

Dataproc's own scheduling is thin: submitting a batch by hand or via the API gives you no built-in cadence, no retry policy, no cross-task lineage, and no alerting. Kestra adds event or cron triggers, automatic retries, full execution history and lineage, and Slack alerts, all expressed as declarative YAML you can version and review. The serverless batch handles the compute; Kestra handles when it runs, what happens on failure, and what runs next.

Prerequisites

• A GCP project with the Dataproc API enabled.
• A service account with permission to submit Dataproc batches.
• A Slack incoming webhook for notifications.

Secrets

• GCP_PROJECT_ID: GCP project hosting Dataproc.
• GCP_SERVICE_ACCOUNT: service account key JSON with Dataproc batch access.
• GCP_REGION: region for the serverless batch (for example, us-central1).
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Set the four secrets above in your Kestra namespace.
2. Replace mainClass, jarFileUris, and args with your own Spark application.
3. Enable the nightly Schedule trigger (or swap it for an event trigger) once configured.
4. Run the flow and confirm the Slack message reports the batch state.

How to extend

• For PySpark, switch to PySparkSubmit with mainPythonFileUri instead of mainClass.
• Chain a downstream BigQuery load after the batch to land the Spark output in the warehouse.
• Replace the Schedule trigger with a file or message event trigger to run on data arrival.
• Add a retry policy on submit_spark to absorb transient cloud errors.

Links

• Plugin docs: https://kestra.io/plugins/plugin-gcp
• Orchestration guide: https://kestra.io/orchestration/dataproc