Dataproc runs the Spark. Kestra runs everything around the submit: trigger it on a schedule or event, gate it on upstream readiness, chain it from GCS to BigQuery, and get per-step retries and one execution history across the whole lakehouse.
Submitting a Spark job is one API call; the pipeline around it is not. Kestra runs Dataproc Serverless batches as steps in a flow: it gates a job on a BigQuery readiness check, chains GCS to Spark to BigQuery, fans batches out over partitions, and adds retries, backfills, and alerting that the Dataproc API alone does not provide. Serverless batches provision and tear down their own compute, so there is no cluster to manage.
Dataproc runs the Spark engine and the serverless compute. Kestra runs the steps around the batch: what triggers it, what it depends on, what it chains to, and how it is retried.
Dataproc Serverless batches provision compute, run, and tear it down per job. Kestra submits io.kestra.plugin.gcp.dataproc.batches.SparkSubmit or PySparkSubmit on a schedule or event, so there is no cluster to size, keep warm, or forget to delete, and cost tracks the job rather than uptime.
Running Spark on missing data wastes compute. Kestra checks BigQuery (or a GCS marker) for fresh rows, then an If submits the batch only when the readiness check passes. This is the dependency Dataproc cannot express: do not burn compute until the upstream data has landed.
The lakehouse chain is multi-step: stage raw files, transform with Spark, load the curated output. Kestra runs PySparkSubmit against GCS data, then bigquery.LoadFromGcs into a gold table, as ordered steps with retries and one execution ID across the chain.
A failed submit has no automatic recovery across the broader pipeline. Kestra retries individual steps with backoff, backfills a dated range, and replays from the failing task, so a transient batch failure does not mean re-running everything by hand or losing the run's history.
Different jobs need different entry points. Kestra submits JVM Spark with SparkSubmit, Python with PySparkSubmit, and SQL scripts with SparkSqlSubmit, each as a typed task, so a mixed-language Spark workload runs from one declarative flow instead of scattered submit scripts.
Patterns data engineering teams run in production today. Each one shows the flow end to end, with the real plugin classes in play.
A Spark job needs to run on a cadence without a standing cluster. Kestra submits it to Dataproc Serverless on a Schedule, which provisions and tears down its own compute per run, and Slack reports the final batch state. Cost tracks the job, not uptime.
Serverless batches provision and tear down compute automatically.
You pay for the batch, not for a cluster sitting idle between runs.
A failed batch retries; the run lands in execution history.
Raw files in GCS need transformation before analytics. Kestra runs a PySpark batch that reads the raw data and writes curated Parquet back to GCS, then loads it into a BigQuery gold table. Microsoft Teams confirms the refresh.
Transform with Spark, then load the curated output to the warehouse.
Debug the Spark step and the load from a single run view.
Trigger on a GCS marker file instead of a schedule.
A nightly batch should not run on a half-loaded source. Kestra queries BigQuery for today's row count, and an If submits the PySpark batch only when the count clears the bar, otherwise it skips and reports. Discord confirms either outcome.
The batch runs only when the source data has actually landed.
A not-ready run reports a skip instead of failing silently.
Swap the count check for a freshness timestamp or marker file.
Heavy SQL transformation runs in Spark, modeling runs in dbt. Kestra submits a Spark SQL batch, then runs dbt build on the result, then validates with a BigQuery query, so the two transformation layers run as one flow with shared run IDs. Email confirms.
Spark SQL and dbt run in order with outputs flowing between them.
A BigQuery check gates anything that reads the result.
The Spark batch and dbt build share one execution for debugging.
Reprocessing many partitions one after another is slow. Kestra fans the submits out with a ForEach, running a PySpark batch per partition in parallel on Dataproc Serverless, then reports completion. A backfill that took hours serially finishes as parallel batches.
ForEach submits a batch per partition concurrently.
Each batch gets its own compute, so parallel runs do not contend.
A concurrency limit caps how many batches run at once.
| Capability | ||||
|---|---|---|---|---|
| Submit Spark on a schedule or event | Native batch tasks + triggers | SubmitJob operator, Python | Custom op, Python | SubmitJob operator, Python |
| Gate a job on upstream readiness | If on a BigQuery check | Sensor + branch, Python | Asset checks, Python | Sensor, Python |
| GCS to Spark to BigQuery in one flow | Native plugins | Operator chain, Python | Assets, Python | Operator chain, Python |
| Serverless, no cluster to manage | batches.* serverless | You wire operators | You manage compute | You wire operators |
| Retries and backfills across the pipeline | Per-task, built in | Yes, Python config | Yes, Python config | Yes, Python config |
| Declarative YAML, no Python | Pure YAML | Python DAGs | Python | Python DAGs |
| Self-hosted or air-gapped option | Open source, self-hosted | GCP-managed | OSS or cloud | Self or managed |
Find answers to your questions right here, and don't hesitate to Contact Us if you couldn't find what you're looking for.
Submit serverless Spark batches, gate them on data readiness, chain GCS to Spark to BigQuery, and fan out over partitions in one flow. Open source, self-hosted, event-driven.
