​A​W​S ​E​C2 with ​Amazon ​R​D​S and ​S3

Install Kestra on AWS EC2 with Postgres RDS database and S3 internal storage backend.

This guide provides instructions for deploying Kestra on Amazon Web Services (AWS). We’ll use EC2 with Docker to host Kestra server, a PostgreSQL RDS database and AWS S3 as storage backend.

Prerequisites:

  • basic knowledge about using a command line interface
  • basic knowledge about EC2, S3 and PostgreSQL.

You can find the corresponding full Terraform configuration in this repository.

Step 1: Create an EC2 instance & install Docker

First, create an EC2 instance. To do so, go to the AWS console and choose EC2.

  1. Give a name to your instance.
  2. Choose Ubuntu as OS.
  3. Instance type: Kestra needs at least 4GiB Memory and 2vCPU to run correctly. Choosing t3-medium is a good starting point.
  4. Create a key-pair to securely connect to your instance. This key is needed to connect through SSH in the following steps.
  5. Create a security group that allows SSH traffic from your IP. Also allow HTTPS traffic.

ec2 creation

ec2 key pair

ec2 network subgroup

You can now click on “Launch instance” and wait a few seconds for the compute to be up and running.

Then you can open a terminal on your laptop and connect to your instance through SSH: ssh -i <your-key-pair.pem> ubuntu@<your-EC2-public-IP>

Kestra can be run directly from the .jar binary or thanks to Docker. We’ll use docker here for quicker setup:

  1. Install Docker on the EC2 instance. You can find the last updated instruction on Docker website.
  2. Install docker-compose.

To check your installation run docker version and docker-compose version. You're now ready to download and launch Kestra server.

Step 2: Download and Run Kestra

Download the official Docker-Compose file: curl -o docker-compose.yml https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml

Use editor such as Vim to modify the docker-compose.yml and set basic authentication to true, and configure your basic authentication credentials to secure your Kestra instance.

yaml
kestra:
  server:
    basic-auth:
      enabled: true
      username: admin
      password: kestra

Then use the following command to start Kestra server: docker-compose up -d

Step 3: Allow external traffic

Kestra is now running and the Kestra server exposes traffic on the 8080 port. To be connect through your web browser, update the inbound traffic rules in the EC2 security group.

Go to the EC2 console, and select Security Group. Choose the security group attached to your EC2 instance and add a new inbound rule to open access to the 8080 port.

If you want to only allow traffic coming from your IP address, set the source to your own IP. If you want to make it open to the entire Internet, leave it at 0.0.0.0.

ec2 inbound rules

You can now access your Kestra instance and create, edit and run Flows.

Step 4: Use AWS RDS Postgres as a database backend

This first installation relies on a Postgres database running alongside the Kestra server - on the EC2 instance (see the postgres service running thanks to the docker-compose).

For a simple PoC, you can keep that Postgres database running in Docker.

However, for a production-grade installation, we recommend a managed database service such as AWS RDS.

Create a AWS RDS database

  1. Go to the RDS console.
  2. Create a database and choose Postgres (Kestra also supports MySQL, but Postgres is recommended)
  3. Set a username and password.
  4. On the connectivity configuration choose “Connect to an EC2 compute resource” and choose your EC2 instance.
  5. Also select the existing DB subnet group and existing VPC security group and choose the one attached to your EC2 instance.
  6. Fine tune instance class, storage type to avoid import AWS costs. For a first step a small Postgres instance is enough
  7. Hit create and wait for completion

rds setup

rds connectivity

Create Kestra database

Before attaching your Kestra server to our new database backend, initialize the database with a base schema as follows:

  1. Connect to your EC2 instance with ssh
  2. Install a PostgresSQL client: sudo apt-get install postgresql-client
  3. Create the Kestra database: createdb -h <your-rds-url-endpoint> -U <your-username> -p 5432 kestra

Update Kestra configuration

In the docker-compose configuration, edit the datasources property of the Kestra service in the following way:

yaml
datasources:
  postgres:
  url: jdbc:postgresql://<your-rds-url-endpoint>:5432/kestra
  driverClassName: org.postgresql.Driver
  username: <your-username>
  password: <your-password>

Because you now use RDS service, you don't need the postgres service anymore. Remove it from the docker-compose.yml file.

In order for the changes to take effect, restart the docker services with docker compose restart or docker compose up -d.

Step 5: Use AWS S3 for storage

By default, internal storage is implemented using the local file system. This section will guide you on how to change the storage backend to S3 to ensure a more reliable, durable and scalable storage.

  1. Go to the S3 console and create a bucket.
  2. Go to IAM and create a new User Group with AWS full access.
  3. Create a new user and attach it to the user group.
  4. For the new user, go to Security Credentials and create an access key. Choose “Application running on an AWS compute service” and retrieve the access and secret keys.
  5. Edit Kestra storage configuration.
yaml
kestra:
  storage:
      type: s3
      s3:
      accessKey: "<your-aws-access-key-id>"
      secretKey: "<your-aws-secret-access-key>"
      region: "<your-aws-region>"
      bucket: "<your-s3-bucket-name>"
  1. Restart docker services.

S3 iam

Next steps

This guide walked you through installing Kestra on an AWS EC2 instance with RDS database and S3 storage backend.

This setup provides the easiest starting point for running Kestra in production on a single machine. For a deployment to a distributed cluster, check the Kubernetes deployment guide.

Reach out via Slack if you encounter any issues or if you have any questions regarding deploying Kestra to production.

Make sure to also check the CI/CD guide to automate your workflow deployments based on changes in Git.