Kubernetes on AWS EKS with Amazon RDS and S3
Deploy Kestra to AWS EKS with a PostgreSQL RDS database and an S3 internal storage backend.
Overview
This guide provides detailed instructions for deploying Kestra to AWS Elastic Kubernetes Service (EKS) with a PostgreSQL RDS database backend and AWS S3 for internal storage.
Prerequisites:
- Basic command-line interface (CLI) skills.
- Familiarity with AWS EKS, RDS, S3, and Kubernetes.
Launch an EKS Cluster
First, install eksctl and kubectl. After installing both, you can create the EKS cluster. There are plenty of configuration options available with eksctl
, but the default settings are sufficient for this guide. Run the following command to create a cluster named my-kestra-cluster
:
eksctl create cluster --name my-kestra-cluster --region us-east-1
Wait for the cluster to be created. Once it is confirmed that the cluster is up and that your kubecontext points to the cluster, run the following command:
kubectl get svc
Launch AWS RDS for PostgreSQL
Navigate to the RDS console to create a PostgreSQL database. Once your database is created, configure the settings, ensuring the database is accessible from your EKS cluster. Make note of the database endpoint and port after creation for later use.
Prepare an AWS S3 Bucket
Create a private S3 bucket (i.e., with public access blocked). Keep a record of the bucket name as this is needed for the Kestra configuration.
Install Kestra on AWS EKS
Add the Kestra Helm chart repository and install Kestra:
helm repo add kestra https://helm.kestra.io/
helm install my-kestra kestra/kestra
In the deployment configuration, integrate RDS and S3 as the database and storage backends, respectively. Set the database connection under datasources
and S3 details under storage
in your Helm values.
Here is how you can configure RDS in the Helm chart's values:
configuration:
kestra:
queue:
type: postgres
repository:
type: postgres
datasources:
postgres:
url: jdbc:postgresql://<your-rds-url-endpoint>:5432/kestra
driverClassName: org.postgresql.Driver
username: <your_username>
password: <your_password>
Also, make sure to disable the PostgreSQL pod by changing the enabled
value in the postgresql
section from true
to false
in the same file.
postgresql:
enabled: false
Add the S3 configuration in the Helm chart's values like in the following example:
configuration:
kestra:
storage:
type: s3
s3:
accessKey: "<your-aws-access-key-id>"
secretKey: "<your-aws-secret-access-key>"
region: "<your-aws-region>"
bucket: "<your-s3-bucket-name>"
Once again, disable the MinIO pod by changing the enabled
value in the minio
section from true
to false
in the same file.
minio:
enabled: false
To apply these configurations, use the following command:
helm upgrade kestra kestra/kestra -f values.yaml
Access Kestra UI
To access the Kestra UI, implement an ingress controller. You can install the AWS Load Balancer (ALB) Controller via Helm:
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
-n kube-system \
--set clusterName=my-kestra-cluster \
--set serviceAccount.create=false \
--set serviceAccount.name=aws-load-balancer-controller
Once the ALB is configured and deployed, access the Kestra UI using the ALB endpoint.
Next steps
This guide walked you through installing Kestra to AWS EKS with PostgreSQL RDS database and S3 storage backend.
Reach out via Slack if you encounter any issues or have questions about deploying Kestra to production.
Was this page helpful?