​Kubernetes on ​A​W​S ​E​K​S with ​Amazon ​R​D​S and ​S3

Deploy Kestra to AWS EKS with a PostgreSQL RDS database and an S3 internal storage backend.

Overview

This guide provides detailed instructions for deploying Kestra to AWS Elastic Kubernetes Service (EKS) with a PostgreSQL RDS database backend and AWS S3 for internal storage.

Prerequisites:

• Basic command-line interface (CLI) skills.
• Familiarity with AWS EKS, RDS, S3, and Kubernetes.

Launch an EKS Cluster

First, install eksctl and kubectl. After installing both, you can create the EKS cluster. There are plenty of configuration options available with eksctl, but the default settings are sufficient for this guide. Run the following command to create a cluster named my-kestra-cluster:

eksctl create cluster --name my-kestra-cluster --region us-east-1

Wait for the cluster to be created. Once it is confirmed that the cluster is up and that your kubecontext points to the cluster, run the following command:

kubectl get svc

Launch AWS RDS for PostgreSQL

Navigate to the RDS console to create a PostgreSQL database. Once your database is created, configure the settings, ensuring the database is accessible from your EKS cluster. Make note of the database endpoint and port after creation for later use.

Prepare an AWS S3 Bucket

Create a private S3 bucket (i.e., with public access blocked). Keep a record of the bucket name as this is needed for the Kestra configuration.

Install Kestra on AWS EKS

Add the Kestra Helm chart repository and install Kestra:

helm repo add kestra https://helm.kestra.io/
helm install my-kestra kestra/kestra

In the deployment configuration, integrate RDS and S3 as the database and storage backends, respectively. Set the database connection under datasources and S3 details under storage in your Helm values.

Here is how you can configure RDS in the Helm chart's values:

configuration:
  kestra:
    queue:
      type: postgres
    repository:
      type: postgres
  datasources:
    postgres:
      url: jdbc:postgresql://<your-rds-url-endpoint>:5432/kestra
      driverClassName: org.postgresql.Driver
      username: <your_username>
      password: <your_password>

Also, make sure to disable the PostgreSQL pod by changing the enabled value in the postgresql section from true to false in the same file.

postgresql:
  enabled: false

Add the S3 configuration in the Helm chart's values like in the following example:

configuration:
  kestra:
    storage:
      type: s3
      s3:
        accessKey: "<your-aws-access-key-id>"
        secretKey: "<your-aws-secret-access-key>"
        region: "<your-aws-region>"
        bucket: "<your-s3-bucket-name>"

Once again, disable the MinIO pod by changing the enabled value in the minio section from true to false in the same file.

minio:
  enabled: false

To apply these configurations, use the following command:

helm upgrade kestra kestra/kestra -f values.yaml

Access Kestra UI

To access the Kestra UI, implement an ingress controller. You can install the AWS Load Balancer (ALB) Controller via Helm:

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
-n kube-system \
--set clusterName=my-kestra-cluster \
--set serviceAccount.create=false \
--set serviceAccount.name=aws-load-balancer-controller

Once the ALB is configured and deployed, access the Kestra UI using the ALB endpoint.

Next steps

This guide walked you through installing Kestra to AWS EKS with PostgreSQL RDS database and S3 storage backend.

Reach out via Slack if you encounter any issues or have questions about deploying Kestra to production.