Clone a GitHub repository and run a Python ETL script

Clone a Git repository and execute a Python ETL script from version-controlled source, with dependencies installed automatically inside an isolated Docker container. This blueprint keeps your transformation logic in Git where it belongs, pulls the latest code at runtime, and runs it in a clean, reproducible Python environment, so your pipeline always reflects what is in your repository without manual packaging or copying scripts onto a server.

How it works

A io.kestra.plugin.core.flow.WorkingDirectory task named python_scripts creates a shared, ephemeral working directory and runs its child tasks in sequence against it:

1. clone_repository uses io.kestra.plugin.git.Clone to clone https://github.com/kestra-io/scripts at the main branch into the working directory.
2. python uses io.kestra.plugin.scripts.python.Commands on a io.kestra.plugin.scripts.runner.docker.Docker task runner. It declares its dependencies (requests, pandas), which are installed into the container, then runs python etl/global_power_plant.py against the freshly cloned code.

What you get

• A working pipeline that pulls Python code from Git and runs it on every execution.
• Reproducible runs in a clean Docker container with pinned dependencies.
• A single shared working directory so the cloned repo and the script run share state.
• No need to bake scripts into images or copy files onto hosts.

Who it's for

• Data engineers running Python ETL from a Git-managed codebase.
• Teams adopting GitOps who want their orchestrator to pull code at runtime.
• Analysts and scientists who maintain transformation scripts in a shared repo.

Why orchestrate this with Kestra

Git itself only stores and versions your code, it has no scheduler, no retries, and no execution engine. Kestra closes that gap: define the whole pipeline as declarative YAML, attach event or schedule triggers, add automatic retries on transient failures, and get full execution history and lineage across the clone and run steps. The Docker task runner guarantees an isolated, reproducible environment so the same commit always runs the same way.

Prerequisites

• A running Kestra instance with Docker available to the worker.
• Network access to GitHub to clone the repository.

Secrets

This blueprint clones a public repository, so no secrets are required. To clone a private repository, add username and password (a personal access token) to the clone_repository task, for example via {{ secret('GITHUB_TOKEN') }}.

Quick start

1. Add this flow to a namespace in your Kestra instance.
2. Confirm the worker can reach GitHub and run Docker containers.
3. Trigger an execution and watch clone_repository then python run in order.
4. Inspect the logs and outputs of the python task.

How to extend

• Point url and branch at your own repository and script path.
• Add or pin more packages in the dependencies list.
• Add a io.kestra.plugin.core.trigger.Schedule trigger to run on a cadence.
• Capture script outputs and pass them to downstream tasks.
• Switch clone_repository to a private repo by adding credentials.

Links

• Git plugin documentation
• Git orchestration with Kestra