How to use the Kubernetes EE plugin

Run Kestra tasks inside Kubernetes pods from Kestra flows.

Authentication

The Kubernetes task runner uses the in-cluster service account by default (no config needed when deployed inside a Kubernetes cluster). For external or multi-cluster use, set config with connection details: masterUrl, oauthToken or username/password, and optional certificate properties (caCertFile, clientCertFile, clientKeyFile, or their -Data variants). Store secrets in secrets and set connection properties on each task.

Task runners

runner.Kubernetes runs task scripts inside a Kubernetes pod. Set namespace (default default) and optionally config for the cluster connection. The runner creates a pod per task execution, streams logs, and deletes the pod on completion (controlled by delete, default true).

Key properties: pullPolicy (default ALWAYS), resume (default true), syncWorkingDirectory (default false), waitUntilRunning (default 10 minutes), waitUntilCompletion (default 1 hour), waitForLogs (maximum extra wait for late logs after containers exit, default 30 seconds — the task returns as soon as the log stream goes quiet, so this is a ceiling, not a fixed delay).

The runner derives a pod-side activeDeadlineSeconds from waitUntilRunning + waitUntilCompletion (or the task timeout) plus a grace period, so kubelet reclaims the pod on its own even if the Worker driving it dies before it can delete or reattach to it. Set podSpec.activeDeadlineSeconds explicitly to override this computed value.

Customize the pod via podSpec (raw Kubernetes pod spec override), containerSpec (container spec override), containerDefaultSpec (settings applied uniformly to all containers, e.g. volumeMounts), and fileSideCarSpec (file sidecar spec override). Configure compute resources via resources.request and resources.limit, each accepting memory and cpu. The file sidecar container uses fileSidecar.image (default busybox).

The Kestra service account running the runner needs RBAC permissions for pods (get, create, delete, watch, list), pods/log (get, watch), and pods/exec (create).

Job mode

Set job.enabled: true to wrap the pod in a batch/v1 Job instead of submitting a raw pod. The Job controller then owns retrying a failed or evicted pod — an eviction is an infrastructure event, not a task/script error — up to job.backoffLimit attempts (default 6, matching the Kubernetes default). Log streaming and file upload/download automatically reattach to whichever pod attempt the Job controller currently has running; resume (Job mode) reattaches to the existing Job rather than a stale pod.

job.podFailurePolicy accepts a raw Pod failure policy passed through as-is, letting the Job controller distinguish infrastructure failures from application failures by exit code or pod condition. It requires the JobPodFailurePolicy feature gate (enabled by default since Kubernetes 1.31) — clusters without it reject the field with a clear error.

Job mode additionally requires RBAC permissions for jobs (get, create, delete, list) and jobs/status (get). parallelism and completions are hardcoded to 1 and are not configurable: this runner only needs the Job controller's retry-on-failure behavior, not its parallel/indexed execution modes.

Detecting a force-deleted/evicted pod can take up to several minutes per attempt, and waitUntilCompletion is a single budget shared across every attempt: size it generously relative to job.backoffLimit so eviction-detection latency alone cannot exhaust it.

Monitoring

Set monitoring.enabled: true to run the task command under kotlp, a ~1 MB portable binary Kestra stages into the working directory and uploads with the task's input files. No image change is needed.

kotlp reports the container's CPU, memory, disk I/O, thread count and open file descriptors as process.* task metrics, sampled every monitoring.metricsInterval (default PT1S), and runs an embedded OTLP receiver so traces the command exports itself are captured too.

It also resolves the long-standing Kubernetes limitation that a pod's stdout and stderr are indistinguishable in its log stream: kotlp tags each line with the stream it came from, so lines the command wrote to stderr are logged at ERROR rather than INFO.

Because the binary is staged as a working-directory file, a task with no input or output files gets the file-transfer init container it would otherwise skip.

The task image must provide a POSIX /bin/sh, so scratch and distroless images do not work with monitoring enabled. kotlp is an Actually Portable Executable bootstrapped through the shell rather than exec'd directly, which is also why it does not depend on the file upload preserving an executable bit.

kotlp unpacks itself on first run into $TMPDIR, falling back to $HOME, so one of those must be writable by the container's user. A pod with readOnlyRootFilesystem: true, or a non-root user on a root-owned working directory, fails with cannot create .../.ape-<version> and exit code 2. To run monitoring on a hardened pod, set fsGroup and point TMPDIR at a writable volume:

podSpec: 
  securityContext: 
    fsGroup: 1000
containerSpec: 
  env: 
    - name: TMPDIR
      value: /kestra/working-dir