​Kubernetes on ​G​C​P ​G​K​E with ​Cloud​S​Q​L and ​Cloud ​Storage

Deploy Kestra to GCP GKE with CloudSQL as a database backend and Google Cloud Storage as internal storage backend.

Overview

This guide provides detailed instructions for deploying Kestra to Google Kubernetes Engine (GKE) with CloudSQL as database backend, and Google Cloud Storage(GCS) for internal storage.

Prerequisites:

• Basic command line interface skills.
• Familiarity with GCP GKE, PostgreSQL, GCS, and Kubernetes.

Launch an GKE Cluster

First, login to GCP using gcloud init.

Run the following command to create an GKE cluster named my-kestra-cluster:

gcloud container clusters create my-kestra-cluster --region=europe-west3

Confirm using the GCP console that the cluster is up.

gke-gcloud-auth-plugin --version

gcloud components install gke-gcloud-auth-plugin

Run the following command to have your kubecontext point to the newly created cluster:

gcloud container clusters get-credentials my-kestra-cluster --region=europe-west3

You can now confirm that your kubecontext points to the GKE cluster using:

kubectl get svc

Install Kestra on GCP GKE

Add the Kestra Helm chart repository and install Kestra:

helm repo add kestra https://helm.kestra.io/
helm install my-kestra kestra/kestra

Workload Identity Setup

If you are using Google Cloud Workload Identity, you can annotate your Kubernetes service account in the Helm chart configuration. This will allow Kestra to automatically use the associated GCP service account for authentication.

To configure this, you can add the following to your "values.yaml" file:

serviceAccount:
  create: true
  name: <your-service-account-name>
  annotations:
    iam.gke.io/gcp-service-account: "<gcp-service-account>@<gcp-project-id>.iam.gserviceaccount.com"

Alternatively, you can apply the annotation directly when you install Kestra using Helm:

helm install my-kestra kestra/kestra \
  --set serviceAccount.annotations.iam.gke.io/gcp-service-account=<gcp-service-account>@<gcp-project-id>.iam.gserviceaccount.com

This configuration links your Kubernetes service account to the GCP service account, enabling Workload Identity for secure access to Google Cloud resources.

Launch CloudSQL

1. Go to the Cloud SQL console.
2. Click on Choose PostgreSQL (Kestra also supports MySQL, but PostgreSQL is recommended).
3. Put an appropriate Instance ID and password for the admin user postgres.
4. Select the latest PostgreSQL version from the dropdown.
5. Choose Enterprise Plus or Enterprise edition based on your requirements.
6. Choose an appropriate preset among Production, Development or Sandbox as per your requirement.
7. Choose the appropriate region and zonal availability.
8. Hit create and wait for completion.

Enable VM connection to database

1. Go to the database overview page and click on Connections from the left-side navigation menu.
2. Go to the Networking tab, and click on Add a Network.
3. In the New Network section, add an appropriate name like Kestra VM, and put your GKE pods IP address range in the Network.
4. Click on Done in the section.
5. Click on Save on the page.

Create database user

1. Go to the database overview page and click on Users from the left-side navigation menu.
2. Click on Add User Account.
3. Put an appropriate username and password, and click on Add.

Create Kestra database

1. Go to the database overview page, and click on Databases from the left side navigation menu.
2. Click on Create Database.
3. Put an appropriate database name, and click on Create.

Update Kestra configuration

Here is how you can configure CloudSQL Database in the Helm chart's values:

configuration:
  kestra:
    queue:
      type: postgres
    repository:
      type: postgres
  datasources:
    postgres:
      url: jdbc:postgresql://<your-db-external-endpoint>:5432/<db_name>
      driverClassName: org.postgresql.Driver
      username: <your-username>
      password: <your-password>

Also, disable the postgres pod by changing enabled value in the postgresql section from true to false in the same file.

postgresql:
  enabled: false

In order for the changes to take effect, run the helm upgrade command as:

helm upgrade my-kestra kestra/kestra -f values.yaml

Prepare a GCS bucket

By default, minio pod is being used as storage backend. This section will guide you on how to change the storage backend to Google Cloud Storage.

By default, internal storage is implemented using the local file system. This section will guide you on how to change the storage backend to Cloud Storage to ensure more reliable, durable, and scalable storage.

1. Go to the Cloud Storage console and create a bucket.
2. Go to IAM and select Service Accounts from the left-side navigation menu.
3. On the Service Accounts page, click on Create Service Account at the top of the page.
4. Put the appropriate Service account name and Service account description, and grant the service account Storage Admin access. Click Done.
5. On the Service Accounts page, click on the newly created service account.
6. On the newly created service account page, go to the Keys tab at the top of the page and click on Add Key. From the dropdown, select Create New Key.
7. Select the Key type as JSON and click on Create. The JSON key file for the service account will get downloaded.
8. We will be using the stringified JSON for our configuration. You can use the bash command % cat <path_to_json_file> | jq '@json' to generate stringified JSON.
9. Edit Kestra storage configuration in the Helm chart's values.

Note: If you want to use a Kubernetes service account configured as a workload identify, you don't need to provide anything for serviceAccount as it will be autodetected for the pod configuration if it's well configured.

configuration:
  kestra:
    storage:
      type: gcs
      gcs:
        bucket: "<your-cloud-storage-bucket-name>"
        projectId: "<your-gcp-project-name>"
        serviceAccount: |
          "<stringified-json-file-contents>"

Also, disable the minio pod by changing enabled value in the minio section from true to false in the same file.

minio:
  enabled: false

In order for the changes to take effect, run the helm upgrade command as:

helm upgrade my-kestra kestra/kestra -f values.yaml

You can validate the storage change from minio to Google Cloud Storage by executing the flow example below with a file and then checking it is uploaded to Google Cloud Storage.

id: inputs
namespace: company.team

inputs:
  - id: file
    type: FILE

tasks:
  - id: validator
    type: io.kestra.plugin.core.log.Log
    message: User {{ inputs.file }}

Commented-out Examples in values.yaml

To provide users with clear guidance on configuring the values.yaml file, we have included some commented-out examples in the configuration. These examples can be used to set up various aspects of Kestra, such as secrets, database configurations, and other key parameters. You can uncomment and modify them as needed.

Here’s an example of how you can define secrets and other configurations in the values.yaml file:

# Example configuration for secrets:
configuration:
  kestra:
    # Configure this section to set secrets for your Kestra instance.
    # secret:
    #   - name: "MY_SECRET_KEY"
    #     value: "my-secret-value"
    #   - name: "ANOTHER_SECRET"
    #     valueFrom:
    #       secretKeyRef:
    #         name: "my-k8s-secret"
    #         key: "my-secret-key"

    # Configure this section to use PostgreSQL as the queue and repository backend.
    # queue:
    #   type: postgres
    # repository:
    #   type: postgres

    # Example of connecting to a PostgreSQL database:
    # datasources:
    #   postgres:
    #     url: jdbc:postgresql://<your-db-endpoint>:5432/<db-name>
    #     driverClassName: org.postgresql.Driver
    #     username: <your-username>
    #     password: <your-password>

# Example to disable default services like MinIO and PostgreSQL if you're using external services:
minio:
  # enabled: false
postgresql:
  # enabled: false

In this example:

-Secrets: You can configure sensitive values as secrets, either hardcoding them or referencing existing Kubernetes secrets.
-Queue and Repository: By default, these can use PostgreSQL or any other supported type. Uncomment the relevant lines to use them.
-PostgreSQL Configuration: Set the datasources section to provide details for connecting to a PostgreSQL database.
-Disabling Services: If you're using external services like CloudSQL or Google Cloud Storage, you can disable the built-in services (MinIO and PostgreSQL).

Feel free to uncomment and modify these examples based on your setup needs. This provides flexibility while keeping your values.yaml well-structured.

Next steps

This guide walked you through installing Kestra to Google GKE with CloudSQL as database and Google Cloud Storage as storage backend.

Reach out via Slack if you encounter any issues or if you have any questions regarding deploying Kestra to production.