​Kubernetes

Install Kestra in a Kubernetes cluster using a Helm chart.

Helm Chart repository

We recommend Kubernetes deployment for production workloads, as it allows you to scale specific Kestra services as needed.

We provide an official Helm Chart to make the deployment easier.

• The chart repository is available under helm.kestra.io.
• The source code of the charts can be found in the kestra-io/helm-charts repository on GitHub.

Install the chart

helm repo add kestra https://helm.kestra.io/
helm install kestra kestra/kestra

By default, the chart will only deploy one standalone Kestra service with one replica. This means that all Kestra server components will be deployed within a single pod. You can change that default behavior and deploy each service independently using the following Helm chart values:

deployments:
  webserver:
    enabled: true
  executor:
    enabled: true
  indexer:
    enabled: true
  scheduler:
    enabled: true
  worker:
    enabled: true
  standalone:
    enabled: false

The chart can also deploy the following related services:

• A Kafka cluster and Zookeeper using kafka.enabled: true
• An Elasticsearch cluster using elasticsearch.enabled: true
• A MinIO standalone using minio.enabled: true
• A PostgreSQL using postgresql.enabled: true

The MinIO (as the internal storage backend) and PostgreSQL (as the database backend) services are enabled by default to provide a fully working setup out of the box.

Configuration

Here is how you can adjust the Kestra configuration:

• Using a Kubernetes ConfigMap via the configuration Helm value.
• Using a Kubernetes Secret via the secrets Helm value.

Both must be valid YAML that will be merged as the Kestra configuration file.

Here is an example showing how to enable Kafka as the queue implementation and configure its bootstrap.servers property using a secret:

configuration:
  kestra:
    queue:
      type: kafka

secrets:
  kestra:
    kafka:
      client:
        properties:
          bootstrap.servers: "localhost:9092"

Docker in Docker (DinD) Worker side car

By default, Docker in Docker (DinD) is installed on the worker in the rootless version. This can be restricted on some environment due to security limitations.

Some solutions you may try:

• On Google Kubernetes Engine (GKE), use a node pool based on UBUNTU_CONTAINERD that works well with Docker DinD, even rootless
• Some Kubernetes clusters support only a root version of DinD; to make your Kestra deployment work, disable the rootless version using the following Helm chart values:

dind:
  image:
    image: docker
    tag: dind
  args:
    - --log-level=fatal

Troubleshooting

Docker in Docker (DinD)

If you face some issues using Docker in Docker e.g. with Script tasks using DOCKER runner, start troubleshooting by attaching the terminal: docker run -it --privileged docker:dind sh. Then, use docker logs container_ID to get the container logs. Also, try docker inspect container_ID to get more information about your Docker container. The output from this command displays details about the container, its environments, network settings, etc. This information can help you identify what might be wrong.

Docker in Docker using Helm charts

On some Kubernetes deployments, using DinD with our default Helm charts can lead to:

Device "ip_tables" does not exist.
ip_tables              24576  4 iptable_raw,iptable_mangle,iptable_nat,iptable_filter
modprobe: can't change directory to '/lib/modules': No such file or directory
error: attempting to run rootless dockerd but need 'kernel.unprivileged_userns_clone' (/proc/sys/kernel/unprivileged_userns_clone) set to 1

To fix this, use root to launch the DinD container by setting the following values:

dind:
  image:
    tag: dind
  args:
    - --log-level=fatal
  securityContext:
    runAsUser: 0
    runAsGroup: 0

securityContext:
  runAsUser: 0
  runAsGroup: 0

Disable Docker in Docker and use Kubernetes task runner

To avoid using root to spin up containers via DinD, disable DinD by setting the following Helm chart values:

dind:
  enabled: false

Then, set Kubernetes task runner as the default way to run script tasks:

pluginDefaults:
  - type: io.kestra.plugin.scripts
    forced: true
    values:
      taskRunner:
        type: io.kestra.plugin.ee.kubernetes.runner.Kubernetes
        # ... your Kubernetes runner configuration