DocsInstallation GuideKubernetes on AWS EKS with Amazon RDS and S3

Kubernetes on AWS EKS with Amazon RDS and S3​Kubernetes on ​A​W​S ​E​K​S with ​Amazon ​R​D​S and ​S3

Deploy Kestra to AWS EKS with PostgreSQL RDS database and S3 internal storage backend.

Overview

This guide provides detailed instructions for deploying Kestra to AWS Elastic Kubernetes Service (EKS) with a PostgreSQL RDS database backend, and AWS S3 for internal storage.

Prerequisites:

  • Basic command line interface skills.
  • Familiarity with AWS EKS, RDS, S3, and Kubernetes.

Launch an EKS Cluster

First, install eksctl and kubectl. After installing them, you can create the EKS cluster. There are plenty of configuration options available with eksctl, but the default settings are sufficient for this guide. Run the following command to create a cluster named my-kestra-cluster:

shell
eksctl create cluster --name my-kestra-cluster --region us-east-1

Wait for the cluster to be created. Then, confirm that the cluster is up, and that your kubecontext points to the cluster:

shell
kubectl get svc

Launch AWS RDS for PostgreSQL

Navigate to the RDS console to create a PostgreSQL database. Configure the settings, ensuring the database is accessible from your EKS cluster. Note the database endpoint and port after creation.

Prepare an AWS S3 Bucket

Create a private S3 bucket (private meaning that public access is blocked). Keep a record of the bucket name as this is needed for the Kestra configuration.

Install Kestra on AWS EKS

Add the Kestra Helm chart repository and install Kestra:

shell
helm repo add kestra https://helm.kestra.io/
helm install my-kestra kestra/kestra

In the deployment configuration, integrate RDS and S3. Set the database connection under datasources and S3 details under storage in your Helm values.

Here is how you can configure RDS in the Helm chart's values:

yaml
configuration:
  kestra:
    queue:
      type: postgres
    repository:
      type: postgres
  datasources:
    postgres:
      url: jdbc:postgresql://<your-rds-url-endpoint>:5432/kestra
      driverClassName: org.postgresql.Driver
      username: your_username
      password: your_password

Also, disable the postgres pod by changing the enabled value in the postgres section from true to false in the same file.

yaml
postgres:
  enabled: false

And here is how you can add the S3 configuration in the Helm chart's values:

yaml
configuration:
  kestra:
    storage:
      type: s3
      s3:
        accessKey: "<your-aws-access-key-id>"
        secretKey: "<your-aws-secret-access-key>"
        region: "<your-aws-region>"
        bucket: "<your-s3-bucket-name>"

Also, disable the minio pod by changing the enabled value in the minio section from true to false in the same file.

yaml
minio:
  enabled: false

Apply these configurations using the following command:

bash
helm upgrade kestra kestra/kestra -f values.yaml

Access Kestra UI

Implement an ingress controller for access. You can install the AWS Load Balancer (ALB) Controller via Helm:

shell
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
-n kube-system \
--set clusterName=my-kestra-cluster \
--set serviceAccount.create=false \
--set serviceAccount.name=aws-load-balancer-controller

Once the ALB is set, you can access the Kestra UI through the ALB URL.

Next steps

This guide walked you through installing Kestra to AWS EKS with PostgreSQL RDS database and S3 storage backend.

Reach out via Slack if you encounter any issues or if you have any questions regarding deploying Kestra to production.