Run a GPU Pod and Collect Its Artifact

Run a GPU workload on Kubernetes and bring its result back into Kestra without writing glue scripts. This blueprint schedules a pod that requests an NVIDIA GPU, runs your container, captures the output file it writes, and posts the artifact location to Slack. It closes the gap between ad hoc kubectl run commands and a governed, repeatable GPU job that has retries, scheduling, and tracked outputs.

How it works

1. The run_gpu_pod task (io.kestra.plugin.kubernetes.core.PodCreate) launches a pod in the gpu-jobs namespace that requests one GPU via the nvidia.com/gpu: 1 resource limit, using the nvidia/cuda runtime image.
2. The container runs its command (here nvidia-smi) and writes results to result.txt in the pod {{ workingDir }}. Kestra pulls that file back into internal storage through outputFiles.
3. The notify task (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) posts a message containing the collected artifact URI from {{ outputs.run_gpu_pod.outputFiles['result.txt'] }}.
4. An optional nightly_job trigger (io.kestra.plugin.core.trigger.Schedule, cron 0 2 * * *) can run the job on a schedule. It ships disabled so you can enable it when ready.

What you get

• A reproducible GPU job defined entirely in declarative YAML.
• Output files collected automatically into Kestra internal storage.
• A Slack message on completion with a link to the artifact.
• A ready scheduled trigger you can flip on for nightly runs.

Who it's for

• ML and data engineers who need to run GPU jobs on Kubernetes on a schedule.
• Platform teams who want governed, auditable GPU workloads instead of manual kubectl runs.
• Researchers who want training or inference outputs captured automatically.

Why orchestrate this with Kestra

Kubernetes schedules pods, but it does not orchestrate a job lifecycle: collecting outputs, retrying on failure, tying runs to upstream events, or notifying downstream systems. Kestra adds event and schedule triggers, automatic retries, full execution lineage, and a single declarative YAML definition. The artifact is captured into internal storage so every run is traceable, something a raw pod spec or a CronJob cannot give you.

Prerequisites

A Kubernetes cluster with GPU nodes and the NVIDIA device plugin installed, plus a Slack incoming webhook.

Secrets

• K8S_MASTER_URL: Kubernetes API server URL.
• K8S_TOKEN: OAuth token for Kubernetes API access.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the secrets above in your Kestra namespace.
2. Replace the container image and command with your own GPU job, writing outputs to {{ workingDir }}.
3. List the files you want returned under outputFiles.
4. Run the flow and check Slack for the artifact link.
5. Enable the nightly_job trigger when you want scheduled runs.

How to extend

• Request multiple GPUs by raising nvidia.com/gpu, or add CPU and memory limits.
• Mount volumes or config maps for datasets and model weights.
• Add downstream tasks to upload the artifact to object storage or a model registry.
• Chain a flow trigger so the GPU job runs when upstream data lands.

Links

• Kubernetes plugin docs
• Kubernetes orchestration with Kestra