​Azure ​V​M with ​Azure ​Database

Install Kestra on Azure VM with Azure Database as a database backend and Blob Storage as internal storage backend.

This guide provides instructions for deploying Kestra on Azure VM with Azure Database as a database backend and Blob Storage as internal storage backend.

Prerequisites:

• basic knowledge about using a command line interface
• basic knowledge about Azure and PostgreSQL.

Create an Azure VM

First, create a virtual machine using Azure Virtual Machines. To do so, go to the Azure portal and choose Virtual Machines.

1. Click on Create and select Azure Virtual Machine.
2. Choose an appropriate Subscription and Resource Group.
3. Give a name for your VM, and choose a Region where it should be launched.
4. For Availability options, choose Availabilty zone, and keep the default availability zone.
5. For Image, choose Ubuntu Server 22.04 LTS - x64 Gen2, and x64 as the VM architecture.
6. Kestra needs at least 4GiB Memory and 2vCPU to run correctly. Choosing the Size as Standard_D2s_v3 is a good starting point.
7. Select SSH public key as the Authentication type.
8. You can keep the default azureuser as the Username.
9. For SSH public key source, you can select Generate new key pair, and provide an appropriate name for the key pair.
10. For Public inbound ports, choose Allow selected ports, and from the Select inbound ports dropdown, select HTTPS and SSH.
11. Click on Review + Create.
12. You can now review the configurations and click on "Create". On the Generate new key pair popup, click on Download private key and create resource.

Wait until the virtual machine is up and running.

Install Docker

In your terminal, run the following commands to SSH into the virtual machine:

chmod 400 <your-key-pair.pem>
ssh -i <your-key-pair.pem> azureuser@<your-VM-public-IP>

Kestra can be started directly from a .jar binary or using Docker. We’ll use Docker here for a quick setup:

1. Install Docker on the Azure VM instance. You can find the last updated instruction on the Docker website.
2. Install docker-compose.

To check your installation, run sudo docker version and sudo docker compose version. You're now ready to download and launch the Kestra server.

Install Kestra

Download the official Docker-Compose file:

curl -o docker-compose.yml \
https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml

Use an editor such as Vim to modify the docker-compose.yml, set basic authentication to true, and configure your basic authentication credentials to secure your Kestra instance.

kestra:
  server:
    basicAuth:
      enabled: true
      username: [email protected] # it must be a valid email address
      password: kestra

Then, use the following command to start the Kestra server:

sudo docker compose up -d

Allow external traffic

Kestra is now running and the Kestra server exposes traffic on the 8080 port. To connect through your web browser, update the inbound traffic rules in the Azure security group.

1. Go to the Virtual Machines console, and select the recently created virtual machine.
2. On the left navigation menu, click on Networking.
3. Under Inbound port rules tab, click on the Add inbound port rule button.
4. In the Add inbound security rule page, put Destination port ranges as 8080. You can keep the default values for the remaining properties. Finally, click on Add at the bottom of the page.

If you want to only allow traffic coming from your local machine, set the Source to your own IP address. To open the instance to the entire Internet, leave it at Any.

You can now access your Kestra instance and start developing flows.

Launch Azure Database

This first installation relies on a PostgreSQL database running alongside the Kestra server - on the VM instance (see the PostgreSQL service running thanks to the docker-compose).

For a simple proof of concept (PoC), you can keep the PostgreSQL database running in Docker.

However, for a production-grade installation, we recommend a managed database service such as Azure Database for PostgreSQL servers.

Launch a database using Azure Database for PostgreSQL servers

1. Go to the Azure Database for PostgreSQL servers.
2. Click on Create Azure Database for PostgreSQL server (Kestra also supports MySQL, but PostgreSQL is recommended).
3. Choose an appropriate Subscription and Resource Group.
4. Put an appropriate Server name, and select the preferred Region.
5. Choose the latest PostgreSQL version. We recommend version 15.
6. Select the Workload type as per your requirement.
7. Choose Authentication method as PostgreSQL authentication only.
8. Provide an appropriate Admin username and Password, and re-write the password in Confirm password.
9. Click on Next: Networking.
10. Click on the select box for Allow public access from any Azure service within Azure to this server.
11. Click on Review + Create. Review the configurations and click on Create.
12. Wait for the database to be provisioned.

Create a Kestra database

1. Go to the database overview page, and click on Databases from the left side navigation menu.
2. Click on Add.
3. Put an appropriate database name, and click on Save at the top.

Update Kestra configuration

In the docker-compose configuration, edit the datasources property of the Kestra service to point Kestra to your Azure database:

datasources:
  postgres:
    url: jdbc:postgresql://<your-db-external-endpoint>:5432/<db_name>
    driverClassName: org.postgresql.Driver
    username: <your-username>
    password: <your-password>

Because you now use the "Azure Database for PostgreSQL servers" service, you don't need the PostgreSQL Docker service anymore. Remove it from the docker-compose.yml file.

In order for the changes to take effect, restart the docker services with sudo docker compose restart or sudo docker compose up -d.

Configure Azure Blob Storage

By default, internal storage is implemented using the local file system. This section will guide you on how to change the storage backend to Blob Storage to ensure more reliable, durable, and scalable storage.

1. Go to the Storage Accounts.
2. Click on Create.
3. Choose an appropriate Subscription and Resource Group.
4. Put an appropriate Storage account name, and select the preferred Region.
5. Select Performance and Redundancy as per your requirement.
6. Click on Review, and post reviewing the configurations, click on Create.
7. Click on the newly created storage account.
8. On the storage account overview page, click on the Containers from the left side navigation menu.
9. Click on Create button at the top to create a new container.
10. Put an appropriate name for the container, and click on Create. A new container will be created.
11. Now, click on Access keys from the left side navigation menu.
12. For one of the keys, either key1 or key2, click on Show for the Connection string, and click on Copy to clipboard button.
13. Note down the connection string with you. We will require this for configuring the storage backend.
14. Edit the Kestra storage configuration in the docker-compose.yml file.

kestra:
  storage:
    type: azure
    azure:
      container: "<your-container>"
      endpoint: "https://<your-storage-account>.blob.core.windows.net/"
      connectionString: "<your-connection-string>"

In order for the changes to take effect, restart the docker services with sudo docker compose restart or sudo docker compose up -d.

For more information on Azure Blob storage configuration, check out the guide here.

Next steps

This guide walked you through installing Kestra on a Azure Virtual Machine with Azure Database for PostgreSQL servers and Azure Blob Storage as storage backend.

This setup provides a simple starting point for running Kestra in production on a single machine. For a deployment to a distributed Kubernetes cluster, check the Azure AKS deployment guide.

Reach out via Slack if you encounter any issues or if you have any questions regarding deploying Kestra to production.

Make sure to also check the CI/CD guide to automate your workflow deployments based on changes in Git.