​Kubernetes on ​Azure ​A​K​S with ​Azure ​Database and ​Blob ​Storage

Deploy Kestra to Azure AKS with Azure Database for PostgreSQL servers as a database backend and Blob Storage as internal storage backend.

Overview

This guide provides detailed instructions for deploying Kestra to Azure Kubernetes Service (AKS) with Azure Database for PostgreSQL servers as database backend, and Blob Storage for internal storage.

Prerequisites:

• Basic command line interface skills.
• Familiarity with Azure AKS, PostgreSQL, Blob Storage, and Kubernetes.

Launch an AKS Cluster

First, login to Azure using az login.

Run the following command to create an AKS cluster named my-kestra-cluster:

az aks create \
  --resource-group <resource-group> \
  --name my-kestra-cluster \
  --enable-managed-identity \
  --node-count 1 \
  --generate-ssh-keys

Confirm that the cluster is up.

Run the following command to have your kubecontext point to the newly created cluster:

az aks get-credentials --resource-group <resource-group> --name my-kestra-cluster

You can now confirm that your kubecontext points to the AKS cluster using:

kubectl get svc

Install Kestra on Azure AKS

Add the Kestra Helm chart repository and install Kestra:

helm repo add kestra https://helm.kestra.io/
helm install my-kestra kestra/kestra

Launch Azure Database for PostgreSQL servers

This first installation relies on a PostgreSQL database running alongside the Kestra server - on a separate pod.

However, for a production-grade installation, we recommend a managed database service such as Azure Database for PostgreSQL servers.

Launch a database using Azure Database for PostgreSQL servers

1. Go to the Azure Database for PostgreSQL servers.
2. Click on Create Azure Database for PostgreSQL server (Kestra also supports MySQL, but PostgreSQL is recommended).
3. Choose an appropriate Subscription and Resource Group.
4. Put an appropriate Server name, and select the preferred Region.
5. Choose the latest PostgreSQL version. We recommend version 15.
6. Select the Workload type as per your requirement.
7. Choose Authentication method as PostgreSQL authentication only.
8. Provide an appropriate Admin username and Password.
9. Click on Next: Networking.
10. Select the box for Allow public access from any Azure service within Azure to this server.
11. Click on Review + Create. Review the configurations and click on Create.
12. Wait for the database to be provisioned.

Create a Kestra database

1. Go to the database overview page, and click on Databases from the left side navigation menu.
2. Click on Add.
3. Put an appropriate database name, and click on Save at the top.

Update Kestra configuration

Here is how you can configure Azure Database in the Helm chart's values:

configuration:
  kestra:
    queue:
      type: postgres
    repository:
      type: postgres
  datasources:
    postgres:
      url: jdbc:postgresql://<your-db-external-endpoint>:5432/<db_name>
      driverClassName: org.postgresql.Driver
      username: <your-username>
      password: <your-password>

Also, disable the PostgreSQL pod by changing enabled value in the postgresql section from true to false in the same file.

postgresql:
  enabled: false

In order for the changes to take effect, run the helm upgrade command as:

helm upgrade my-kestra kestra/kestra -f values.yaml

Prepare an Azure Blob Storage container

By default, Minio pod is being used as storage backend. This section will guide you on how to change the storage backend to Blob Storage.

1. Go to the Storage Accounts.
2. Click on Create.
3. Choose an appropriate Subscription and Resource Group.
4. Put an appropriate Storage account name, and select the preferred Region.
5. Select Performance and Redundancy as per your requirement.
6. Click on Review, and post reviewing the configurations, click on Create.
7. Click on the newly created storage account.
8. On the storage account overview page, click on the Containers from the left side navigation menu.
9. Click on Create button at the top to create a new container.
10. Put an appropriate name for the container, and click on Create. A new container will be created.
11. Now, click on Access keys from the left side navigation menu.
12. For one of the keys, either key1 or key2, click on Show for the Connection string, and click on Copy to clipboard button.
13. Note down the connection string with you. We will require this for configuring the storage backend.
14. Here is how you can add Blob Storage configuration in the Helm chart's values:

configuration:
  kestra:
    storage:
      type: azure
      azure:
        container: "<your-container>"
        endpoint: "https://<your-storage-account>.blob.core.windows.net/"
        connectionString: "<your-connection-string>"

Also, disable the Minio pod by changing the enabled value in the minio section from true to false in the same file.

minio:
  enabled: false

In order for the changes to take effect, run the helm upgrade command as:

helm upgrade my-kestra kestra/kestra -f values.yaml

Access Kestra UI

Implement an ingress controller for access. You can install AKS Load Balancer Controller via Helm:

helm install aks-load-balancer-controller application-gateway-kubernetes-ingress/ingress-azure \
     --set appgw.name=kestra-application-gateway \
     --set appgw.resourceGroup=<resource-group> \
     --set appgw.subscriptionId=<subscription-uuid> \
     --set appgw.shared=false \
     --set armAuth.type=servicePrincipal \
     --set armAuth.secretJSON=$(az ad sp create-for-rbac --role Contributor --scopes /subscriptions/<subscription-uuid>/resourceGroups/<resource-group> --sdk-auth | base64 -w0) \
     --set rbac.enabled=true \
     --set verbosityLevel=3 \
     --set kubernetes.watchNamespace=default \
     --set aksClusterConfiguration.apiServerAddress=<aks-server-address>

Once the load balancer is set, you can access the Kestra UI through the ALB URL.

Next steps

This guide walked you through installing Kestra to Azure AKS with Azure Database for PostgreSQL as database and Blob Storage as storage backend.

Reach out via Slack if you encounter any issues or if you have any questions regarding deploying Kestra to production.