Clone a GitHub Repo and Run Its Shell and Python Scripts

Keep pipeline logic in Git and let the orchestrator fetch it at runtime. This Kestra blueprint clones a private GitHub repository into a shared working directory, produces a sample CSV, then runs the repository's own bash and Python scripts against that file, complete with pinned Python dependencies and task caching. Your scripts stay code-reviewed and version-controlled; Kestra supplies the execution environment, data passing, and observability.

How it works

1. The my_working_dir task (io.kestra.plugin.core.flow.WorkingDirectory) opens one shared filesystem for all child tasks, so the clone output is directly visible to the script runs.
2. The clone_git task (io.kestra.plugin.git.Clone) checks out your repository at the branch given by the branch input (default main), authenticating with a username plus a GITHUB_TOKEN secret for private repos.
3. The create_csv task (io.kestra.plugin.core.storage.Write) writes a small users CSV to internal storage and exposes its URI.
4. The shell_task (io.kestra.plugin.scripts.shell.Commands) marks the repo's pre-process/check.sh executable and runs it with the CSV URI as its argument; Kestra transparently materializes internal storage URIs referenced in commands as local files.
5. The python_task (io.kestra.plugin.scripts.python.Commands) runs pre-process/pre_process.py in a python:3.13-slim container with pandas and click installed via dependencies, and taskCache enabled so repeat runs skip re-installing.

Adjust the two script paths to match your repository layout; any executable script works.

What you get

• Scripts executed straight from source control, traceable to a branch and commit.
• A single working directory that wires clone output into every downstream task, following the one-working-dir-per-flow best practice.
• Mixed-language execution (bash plus Python) sharing one filesystem and one data file.
• Dependency pinning and caching for the Python step without any requirements.txt shipping ceremony.

Who it's for

• Data engineers whose transformation and validation scripts already live in GitHub.
• Platform teams enforcing code review on anything that touches production data.
• Teams replacing cron plus git-pull wrapper scripts with observable orchestration.

Why orchestrate this with Kestra

A cron job that does git clone && ./run.sh gives you no logs, no retries, no input parameters, and no record of which branch ran. Kestra makes each step a task: the clone is declarative and authenticated via secrets, the branch is a runtime input, script output is captured per task, and internal storage moves data between steps without shared volumes. WorkingDirectory scopes the filesystem so parallel executions never collide, and the whole definition is YAML you can review and version alongside the scripts it runs.

Prerequisites

• A GitHub repository containing your scripts (the flow expects pre-process/check.sh and pre-process/pre_process.py; adjust paths to your layout).
• Docker available to the worker for the Python container image.

Secrets

• GITHUB_TOKEN: a GitHub personal access token with read access to the repository. Public repositories can omit username and password on the clone_git task entirely.

Quick start

1. Add the GITHUB_TOKEN secret to your Kestra namespace.
2. Edit clone_git with your repository URL and username, and point the two commands at real script paths in your repo.
3. Import the flow into company.team and execute it, optionally overriding the branch input to test a feature branch.
4. Inspect each task's logs to see the shell check and the pandas-based validation run against the generated CSV.

How to extend

• Replace create_csv with a real extract task (S3 download, HTTP request, SQL query) and feed its URI to the scripts.
• Add a io.kestra.plugin.core.trigger.Schedule trigger to run the scripts nightly, or a webhook to run on demand.
• Fan out over many files with ForEach calling a subflow that owns its own working directory.
• Sync the repository once per namespace with io.kestra.plugin.git.SyncNamespaceFiles instead of cloning per execution when many flows share the same scripts.

Links

• Git plugin documentation: https://kestra.io/plugins/plugin-git
• Version control with Git: https://kestra.io/docs/version-control-cicd