docsinstallation

GCP VM with Cloud SQL and GCS​G​C​P ​V​M with ​Cloud ​S​Q​L and ​G​C​S

Install Kestra on a GCP VM with Cloud SQL as the database backend and Cloud Storage as the internal storage backend.

Overview

This guide provides instructions for deploying Kestra on Google Cloud Platform (GCP). We use Compute Engine with Docker to host the Kestra server, a Cloud SQL PostgreSQL database, and Cloud Storage as the storage backend.

Prerequisites:

  • Basic command-line interface (CLI) skills.
  • Familiarity with Compute Engine, Cloud Storage, and PostgreSQL.

Create a VM instance

First, create a VM instance using the Compute Engine. To do so, go to the GCP console and choose Compute Engine.

  1. Click the Create Instance button at the top.
  2. Give a name to your instance.
  3. Choose an appropriate Region and Zone.
  4. Choose the General Purpose machine of the E2 series.
  5. Machine type: Kestra requires at least 4GiB of memory and 2 vCPUs to run correctly. Choosing the Preset machine type e2-standard-2 is a good starting point.
  6. Click on Change in the Boot Disk section, as we would like to change the image.
  7. Under the "Public Images" tab, choose Ubuntu as the operating system and the Ubuntu 22.04 LTS version.
  8. Continue with the Allow default access access scope, and select Allow HTTPS traffic in the Firewall section.

vm creation_1

vm creation 2

change_boot_disk_image

vm_creation_3

You can now click on Create and wait a few seconds for the VM instance to be up and running.

Install Docker

Click on the SSH button on the right side of the VM instance details to SSH into the VM instance terminal. Click on the Authorize button in the pop-up to authorize the SSH connection into the VM instance.

ssh_into_vm

Kestra can be started directly from a .jar binary or using Docker. We use Docker here for a quicker setup. Install Docker on the GCP VM instance. You can find the last updated instruction on the Docker website.

To check your installation, run sudo docker version and sudo docker compose version. You're now ready to download and launch the Kestra server.

Install Kestra

Download the official Docker-Compose file:

bash
curl -o docker-compose.yml \
https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml

Use an editor such as Vim to modify the docker-compose.yml, set basic authentication to true, and configure your basic authentication credentials to secure your Kestra instance.

yaml
kestra:
  server:
    basicAuth:
      enabled: true
      username: [email protected] # it must be a valid email address
      password: kestra

vm_network_details_option

vm_network_interface_details

vm_firewall_policies

vm_create_firewall_rule

You can now access your Kestra instance and start developing flows.

Launch Cloud SQL

This first installation relies on a PostgreSQL database running alongside the Kestra server - on the VM instance (see the PostgreSQL service running thanks to the docker-compose).

For a simple proof of concept (PoC), you can keep the PostgreSQL database running in Docker.

However, for a production-grade installation, we recommend a managed database service such as Cloud SQL.

Create a Cloud SQL database

  1. Go to the Cloud SQL console.
  2. Click on Choose PostgreSQL (Kestra also supports MySQL, but PostgreSQL is recommended).
  3. Put an appropriate Instance ID and password for the admin user postgres.
  4. Select the latest PostgreSQL version from the dropdown.
  5. Choose Enterprise Plus or Enterprise edition based on your requirements.
  6. Choose an appropriate preset among Production, Development or Sandbox as per your requirement.
  7. Choose the appropriate region and zonal availability.
  8. Expand Show Show Configuration Options at the bottom of the page.

db_choices

db_setup

db_show_config

Enable VM connection to database

  1. Expand the Connections section from the dropdown.
  2. Uncheck Public IP and check Private IP. If this is your first time using a Private IP connection, you will be prompted to Setup Connection.
  3. You will then need to choose Enable API on the right hand side pop out.
  4. Choose Use an automatically allocated IP range and click Continue.
  5. Click on Create Connection.

db_connections

db_enable_api

db_auto_allocate

db_auto_allocate

Enable Deletion

If you are just testing or would like to be able to delete your instance and all of its data, then expand the Data Protection on the left hand side and make sure Enable deletion protection is UNCHECKED.

db_deletion_protection

Create database user

  1. Go to the database overview page and click Users from the left-side navigation menu.
  2. Click Add User Account.
  3. Put an appropriate username and password and click Add.

db_users

db_user_creation

Create Kestra database

  1. Go to the database overview page and click Databases from the left side navigation menu.
  2. Click Create Database.
  3. Put an appropriate database name and click Create.

Update Kestra configuration

In the docker-compose configuration, edit the datasources property of the Kestra service in the following way:

yaml
datasources:
  postgres:
    url: jdbc:postgresql://<your-db-external-endpoint>:5432/<db_name>
    driverClassName: org.postgresql.Driver
    username: <your-username>
    password: <your-password>

And delete the depends_on section at the end of the YAML file:

yaml
depends_on:
  postgres:
    condition: service_started

Since you're now using Cloud SQL, you no longer need the PostgreSQL Docker service. Remove it from the docker-compose.yml file.

In order for the changes to take effect, restart the docker services with sudo docker compose restart or sudo docker compose up -d.

Configure GCS

By default, internal storage is implemented using the local file system. This section guides you how to change the storage backend to Cloud Storage to ensure more reliable, durable, and scalable storage.

  1. Go to the Cloud Storage console and create a bucket.
  2. Go to IAM and select Service Accounts from the left-side navigation menu.
  3. On the Service Accounts page, click on Create Service Account at the top of the page.
  4. Put the appropriate Service account name and Service account description and grant the service account Storage Admin access. Click Done.
  5. On the Service Accounts page, click on the newly created service account.
  6. On the newly created service account page, go to the Keys tab at the top of the page and click on Add Key. From the dropdown, select Create New Key.
  7. Select the Key type as JSON and click Create. The JSON key file for the service account will get downloaded.
  8. We will be using the stringified JSON for our configuration. You can use the bash command cat <path_to_json_file> | jq '@json' to generate stringified JSON.
  9. Edit the Kestra storage configuration.
yaml
kestra:
  storage:
    type: gcs
    gcs:
      bucket: "<your-cloud-storage-bucket-name>"
      projectId: "<your-gcp-project-name>"
      serviceAccount: |
                      <stringified-json-file-contents>

In order for the changes to take effect, restart the docker services with sudo docker compose restart or sudo docker compose up -d.

Next steps

This guide walked you through installing Kestra on a GCP VM instance with a Cloud SQL database and Cloud Storage as the storage backend.

This setup provides the easiest starting point for running Kestra in production on a single machine. For a deployment to a distributed cluster on GCP, check the GKE Kubernetes deployment guide.

Reach out via Slack if you encounter any issues or have any questions regarding deploying Kestra to production.

Also, check out the CI/CD guide to automate your workflow deployments based on changes in Git.

Was this page helpful?