Clone a Git repository with PySpark code and run a Spark job

Pull your PySpark code straight from version control and submit it to a Spark cluster in one declarative workflow. This blueprint clones a Git repository containing your Spark application, then runs the job with the Spark Submit CLI, so your distributed Python workloads stay in sync with the code in your repo instead of manually copied JARs or scripts. It is a clean pattern for GitOps-style Spark deployments where the latest committed code is always what runs.

How it works

The flow runs inside a single io.kestra.plugin.core.flow.WorkingDirectory task so both steps share the same isolated working directory:

1. clone_repository uses io.kestra.plugin.git.Clone to clone the configured url (the kestra-io/scripts repository) at the main branch into the working directory.
2. spark_job uses io.kestra.plugin.spark.SparkCLI to run spark-submit, pointing at the Spark master with --master spark://localhost:7077 and executing the cloned PySpark file (etl/spark_pi.py).

Because both tasks live in the same WorkingDirectory, the freshly cloned code is immediately available to the spark-submit command without any extra file shuffling.

What you get

• Spark jobs that always run the exact code committed to your Git repository.
• A reproducible clone-then-submit pattern in a single workflow.
• Shared working directory so cloned files flow directly into the Spark job.
• A starting point for distributed PySpark ETL on a standalone Spark cluster.

Who it's for

• Data engineers running PySpark ETL and analytics jobs.
• Platform teams adopting GitOps for Spark application deployment.
• Teams that want code review and version history to gate what Spark executes.

Why orchestrate this with Kestra

Spark's own scheduler distributes tasks within a job, but it does not pull your code from Git, manage retries across the whole pipeline, or react to upstream events. Kestra wraps the clone and submit steps in declarative YAML, adds event and schedule triggers, retries, and full execution lineage, and lets you chain Spark jobs into larger data pipelines that Spark alone cannot coordinate.

Prerequisites

• A reachable Spark master with the cluster port exposed (this flow targets spark://localhost:7077).
• The Spark plugin and a Spark runtime available to the worker running spark-submit.
• Network access from Kestra to the Git repository being cloned.

Secrets

This blueprint clones a public repository and submits to a local Spark master, so it references no {{ secret('NAME') }} values. To pull from a private repository, add a Git token and pass it to the Clone task as username and password using a secret such as {{ secret('GITHUB_TOKEN') }}.

Quick start

1. Add this flow to a namespace in your Kestra instance.
2. Make sure your Spark master is running and the port 7077 is exposed.
3. Update the url, branch, and the spark-submit command to match your own repository and PySpark application.
4. Execute the flow and watch the clone and Spark job run in the shared working directory.

How to extend

• Point url at your own repository and submit your real PySpark application.
• Swap --master spark://localhost:7077 for a remote master, YARN, or Kubernetes target.
• Pass --conf and --py-files arguments to spark-submit for tuning and dependencies.
• Add a schedule or event trigger to run the job on new commits or on a cadence.
• Chain downstream tasks to load Spark output into a warehouse or notify on completion.

Links

• Spark plugin docs
• Git plugin docs
• Git orchestration with Kestra