​Kubernetes on ​A​W​S ​E​K​S with ​Amazon ​R​D​S and ​S3

Deploy Kestra to AWS EKS with PostgreSQL RDS database and S3 internal storage backend.

Overview

This guide provides detailed instructions for deploying Kestra to AWS Elastic Kubernetes Service (EKS) with a PostgreSQL RDS database backend and AWS S3 for internal storage.

Prerequisites:

• Basic command line interface skills.
• Familiarity with AWS EKS, RDS, S3, and Kubernetes.

Launch an EKS Cluster

First, install eksctl and kubectl. After installing both, you can create the EKS cluster. There are plenty of configuration options available with eksctl, but the default settings are sufficient for this guide. Run the following command to create a cluster named my-kestra-cluster:

eksctl create cluster --name my-kestra-cluster --region us-east-1

Wait for the cluster to be created. Once it is confirmed that the cluster is up and that your kubecontext points to the cluster, run the following command:

kubectl get svc

Launch AWS RDS for PostgreSQL

Navigate to the RDS console to create a PostgreSQL database. Once your database is created, configure the settings, ensuring the database is accessible from your EKS cluster. Make note the database endpoint and port after creation for later use.

Prepare an AWS S3 Bucket

Create a private S3 bucket (private meaning that public access is blocked). Keep a record of the bucket name as this is needed for the Kestra configuration.

Install Kestra on AWS EKS

Add the Kestra Helm chart repository and install Kestra:

helm repo add kestra https://helm.kestra.io/
helm install my-kestra kestra/kestra

In the deployment configuration, integrate RDS and S3. Set the database connection under datasources and S3 details under storage in your Helm values.

Here is how you can configure RDS in the Helm chart's values:

configuration:
  kestra:
    queue:
      type: postgres
    repository:
      type: postgres
  datasources:
    postgres:
      url: jdbc:postgresql://<your-rds-url-endpoint>:5432/kestra
      driverClassName: org.postgresql.Driver
      username: your_username
      password: your_password

Also, make sure to disable the PostgreSQL pod by changing the enabled value in the postgresql section from true to false in the same file.

postgresql:
  enabled: false

Add the S3 configuration in the Helm chart's values like in the following example:

configuration:
  kestra:
    storage:
      type: s3
      s3:
        accessKey: "<your-aws-access-key-id>"
        secretKey: "<your-aws-secret-access-key>"
        region: "<your-aws-region>"
        bucket: "<your-s3-bucket-name>"

Once again, disable the MinIO pod by changing the enabled value in the minio section from true to false in the same file.

minio:
  enabled: false

To apply these configurations, use the following command:

helm upgrade kestra kestra/kestra -f values.yaml

Access Kestra UI

To access the Kestra UI, implement an ingress controller. You can install the AWS Load Balancer (ALB) Controller via Helm:

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
-n kube-system \
--set clusterName=my-kestra-cluster \
--set serviceAccount.create=false \
--set serviceAccount.name=aws-load-balancer-controller

Once the ALB is set, you can access the Kestra UI through the ALB URL.

Next steps

This guide walked you through installing Kestra to AWS EKS with PostgreSQL RDS database and S3 storage backend.

Reach out via Slack if you encounter any issues or if you have any questions regarding deploying Kestra to production.