Run a Python script in a Kubernetes pod

Run a Python script on a remote Kubernetes pod directly from a Kestra flow, with full control over CPU and memory requests so each job lands on the right node and stays inside its resource budget. This blueprint solves a common gap in container orchestration: you want ephemeral, isolated compute for data and automation scripts without writing your own pod manifests, managing the pod lifecycle, or shipping files in and out of the container by hand. Kestra handles pod creation, resource sizing, file exchange, and cleanup while you keep your logic in plain Python.

How it works

The flow has a single task, send_data, of type io.kestra.plugin.scripts.python.Script. Instead of running on the local worker, the script is dispatched to a Kubernetes pod through the io.kestra.plugin.ee.kubernetes.runner.Kubernetes task runner.

• The task runner connects to your cluster using masterUrl, username, and certificate data passed through config, and schedules the pod in the default Kubernetes namespace with pullPolicy: ALWAYS.
• resources.request reserves 500m CPU and 128Mi memory for the pod.
• The Python script imports kestra, prints host, Python version, platform, and OS/arch details, and writes them both to Kestra.outputs(...) and to environment_info.json.
• outputFiles captures *.json, so a sidecar container downloads environment_info.json back into Kestra storage. The kestra pip dependency is installed via dependencies.

If your script task adds inputFiles or namespaceFiles, an init container uploads them into the main container; with outputFiles, a sidecar downloads results. All containers share an in-memory emptyDir volume for file exchange. If a task is resubmitted after a retry or worker crash, the new worker reattaches to the already running or finished pod instead of starting a new one.

What you get

• Ephemeral, isolated Python execution on Kubernetes per run.
• Declarative CPU and memory requests defined in YAML.
• Automatic file upload and download between Kestra and the pod.
• Structured outputs available to downstream tasks.

Who it's for

• Data and platform engineers running Python jobs on existing clusters.
• Teams that want resource-isolated tasks without hand-written manifests.
• DevOps engineers standardizing script execution across environments.

Why orchestrate this with Kestra

Kubernetes schedules pods, but it does not orchestrate multi-step data pipelines. Kestra adds event and schedule triggers, automatic retries, run-level lineage, and a fully declarative YAML definition, then offloads heavy work to Kubernetes pods sized exactly as you specify. You get pod-level isolation and resource control without leaving your workflow layer, and without building CronJobs and glue scripts the cluster scheduler alone cannot coordinate.

Prerequisites

• A reachable Kubernetes cluster and a Kestra Enterprise Edition instance.
• A container image with Python (the kestra package is installed at runtime via dependencies).

Secrets

Store these as Kestra secrets:

• K8S_USERNAME: Kubernetes user name.
• MASTER_URL: cluster API server URL.
• caCertData: cluster CA certificate data.
• clientCertData: client certificate data.
• clientKeyData: client private key data.

Quick start

1. Add the five secrets above to your Kestra instance.
2. Copy this blueprint into a flow and set the Kubernetes namespace to your target.
3. Adjust resources.request to match your workload.
4. Execute the flow and inspect the outputs and the captured environment_info.json.

How to extend

• Replace the demo script with your own Python logic and pip dependencies.
• Add inputFiles or namespaceFiles to ship code or config into the pod.
• Add resources.limit alongside request to cap usage.
• Add a schedule or event trigger to run the job automatically.

Links

• Plugin docs: https://kestra.io/plugins/plugin-kubernetes
• Orchestrate Kubernetes with Kestra: https://kestra.io/orchestration/kubernetes