GCP VM with Cloud SQL and GCS
Install Kestra on a GCP VM with Cloud SQL as the database backend and Cloud Storage as the internal storage backend.
Overview
This guide provides instructions for deploying Kestra on Google Cloud Platform (GCP). We use Compute Engine with Docker to host the Kestra server, a Cloud SQL PostgreSQL database, and Cloud Storage as the storage backend.
Prerequisites:
- Basic command-line interface (CLI) skills.
- Familiarity with Compute Engine, Cloud Storage, and PostgreSQL.
Create a VM instance
First, create a VM instance using the Compute Engine. To do so, go to the GCP console and choose Compute Engine.
- Click the Create Instance button at the top.
- Give a name to your instance.
- Choose an appropriate Region and Zone.
- Choose the General Purpose machine of the E2 series.
- Machine type: Kestra requires at least 4GiB of memory and 2 vCPUs to run correctly. Choosing the Preset machine type e2-standard-2 is a good starting point.
- Click on Change in the Boot Disk section, as we would like to change the image.
- Under the "Public Images" tab, choose Ubuntu as the operating system and the Ubuntu 22.04 LTS version.
- Continue with the Allow default access access scope, and select Allow HTTPS traffic in the Firewall section.
You can now click on Create and wait a few seconds for the VM instance to be up and running.
Install Docker
Click on the SSH button on the right side of the VM instance details to SSH into the VM instance terminal. Click on the Authorize button in the pop-up to authorize the SSH connection into the VM instance.
Kestra can be started directly from a .jar
binary or using Docker. We use Docker here for a quicker setup.
Install Docker on the GCP VM instance. You can find the last updated instruction on the Docker website.
To check your installation, run sudo docker version
and sudo docker compose version
. You're now ready to download and launch the Kestra server.
Install Kestra
Download the official Docker-Compose file:
curl -o docker-compose.yml \
https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml
Use an editor such as Vim to modify the docker-compose.yml
, set basic authentication to true
, and configure your basic authentication credentials to secure your Kestra instance.
kestra:
server:
basicAuth:
enabled: true
username: [email protected] # it must be a valid email address
password: kestra
Note that if you haven't set up basic authentication in the previous step, your Kestra instance will be publicly accessible to anyone without authentication.
You can now access your Kestra instance and start developing flows.
Launch Cloud SQL
This first installation relies on a PostgreSQL database running alongside the Kestra server - on the VM instance (see the PostgreSQL service running thanks to the docker-compose).
For a simple proof of concept (PoC), you can keep the PostgreSQL database running in Docker.
However, for a production-grade installation, we recommend a managed database service such as Cloud SQL.
Create a Cloud SQL database
- Go to the Cloud SQL console.
- Click on
Choose PostgreSQL
(Kestra also supports MySQL, but PostgreSQL is recommended). - Put an appropriate Instance ID and password for the admin user
postgres
. - Select the latest PostgreSQL version from the dropdown.
- Choose
Enterprise Plus
orEnterprise
edition based on your requirements. - Choose an appropriate preset among
Production
,Development
orSandbox
as per your requirement. - Choose the appropriate region and zonal availability.
- Expand Show
Show Configuration Options
at the bottom of the page.
Enable VM connection to database
- Expand the
Connections
section from the dropdown. - Uncheck Public IP and check Private IP. If this is your first time using a Private IP connection, you will be prompted to
Setup Connection
. - You will then need to choose
Enable API
on the right hand side pop out. - Choose
Use an automatically allocated IP range
and clickContinue
. - Click on
Create Connection
.
Enable Deletion
If you are just testing or would like to be able to delete your instance and all of its data, then expand the Data Protection
on the left hand side and make sure Enable deletion protection
is UNCHECKED.
Create database user
- Go to the database overview page and click Users from the left-side navigation menu.
- Click Add User Account.
- Put an appropriate username and password and click Add.
Create Kestra database
- Go to the database overview page and click Databases from the left side navigation menu.
- Click Create Database.
- Put an appropriate database name and click Create.
Update Kestra configuration
In the docker-compose configuration, edit the datasources
property of the Kestra service in the following way:
datasources:
postgres:
url: jdbc:postgresql://<your-db-external-endpoint>:5432/<db_name>
driverClassName: org.postgresql.Driver
username: <your-username>
password: <your-password>
And delete the depends_on
section at the end of the YAML file:
depends_on:
postgres:
condition: service_started
Since you're now using Cloud SQL, you no longer need the PostgreSQL Docker service. Remove it from the docker-compose.yml
file.
In order for the changes to take effect, restart the docker services with sudo docker compose restart
or sudo docker compose up -d
.
Configure GCS
By default, internal storage is implemented using the local file system. This section guides you how to change the storage backend to Cloud Storage to ensure more reliable, durable, and scalable storage.
- Go to the Cloud Storage console and create a bucket.
- Go to IAM and select Service Accounts from the left-side navigation menu.
- On the Service Accounts page, click on Create Service Account at the top of the page.
- Put the appropriate Service account name and Service account description and grant the service account Storage Admin access. Click Done.
- On the Service Accounts page, click on the newly created service account.
- On the newly created service account page, go to the Keys tab at the top of the page and click on Add Key. From the dropdown, select Create New Key.
- Select the Key type as JSON and click Create. The JSON key file for the service account will get downloaded.
- We will be using the stringified JSON for our configuration. You can use the bash command
cat <path_to_json_file> | jq '@json'
to generate stringified JSON. - Edit the Kestra storage configuration.
kestra:
storage:
type: gcs
gcs:
bucket: "<your-cloud-storage-bucket-name>"
projectId: "<your-gcp-project-name>"
serviceAccount: |
<stringified-json-file-contents>
In order for the changes to take effect, restart the docker services with sudo docker compose restart
or sudo docker compose up -d
.
Next steps
This guide walked you through installing Kestra on a GCP VM instance with a Cloud SQL database and Cloud Storage as the storage backend.
This setup provides the easiest starting point for running Kestra in production on a single machine. For a deployment to a distributed cluster on GCP, check the GKE Kubernetes deployment guide.
Reach out via Slack if you encounter any issues or have any questions regarding deploying Kestra to production.
Also, check out the CI/CD guide to automate your workflow deployments based on changes in Git.
Was this page helpful?