Configure Google Service Account​Configure ​Google ​Service ​Account

Setup a Google Service Account inside of Kestra.

When you're using Google Cloud (and for some Google Workspace apps), you're going to need to authenticate inside of Kestra. The best way to do this is by using a Service Account. However, there's a few ways you can set this up. This guide will walk you through the best way to get your service account working correctly inside of Kestra.

Create Service Account inside of Google Cloud

Inside of Google Cloud, head to IAM and then Service Accounts. In here you can add the specific roles to the service account before creating it (this will depend on your use case).

Once you've done that, you can go to Keys and click on Add Key. From the dropdown, select Create New Key. Select the Key type as JSON and click on Create. Download this as we'll need this in a second.

For more information on Google Cloud Service Accounts, check out the documentation.

Configuring a task with a Service Account

Inside of Kestra, you can paste the service account JSON directly to the task property. This is useful for testing purposes:

yaml
- id: upload
  type: io.kestra.plugin.googleworkspace.drive.Upload
  from: "{{ inputs.file }}"
  parents:
    - "1HuxzpLt1b0111MuKMgy8wAv-m9Myc1E_"
  name: "My awesome CSV"
  contentType: "text/csv"
  mimeType: "application/vnd.google-apps.spreadsheet"
  serviceAccount: |
    {
      "type": "service_account",
      "project_id": "...",
      "private_key_id": "...",
      "private_key": "...",
      "client_email": "...",
      "client_id": "...",
      "auth_uri": "https://accounts.google.com/o/oauth2/auth",
      "token_uri": "https://oauth2.googleapis.com/token",
      "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
      "client_x509_cert_url": "...",
      "universe_domain": "googleapis.com"
    }

Add Service Account as a Secret

We can add our Service Account with the serviceAccount property to any of our Google Cloud or Workspaces tasks. To do this, we'll need to add it as a secret to Kestra. There's a number of ways to add secrets, but we're going to add it via environment variables which will link to our Docker Compose file. If you want more information regarding how secrets work, check out the secrets page.

Once you have the service account file downloaded, you can rename it to service-account.json. Then we'll encode the service account JSON and store it inside of a file named .env_encoded which will hold all of our encoded secrets:

bash
echo SECRET_GCP_SERVICE_ACCOUNT=$(cat service-account.json | base64 -w 0) >> .env_encoded

If you already have an existing .env file, you can use the following bash script:

bash
#!/bin/bash

ENV_FILENAME=.env_encoded

while IFS='=' read -r key value; do
  echo "SECRET_$key=$(echo -n "$value" | base64)";
done < .env > $ENV_FILENAME

# Encodes the service account file without line wrapping to make sure the whole JSON value is intact.
echo "SECRET_GCP_SERVICE_ACCOUNT=$(cat service-account.json | base64 -w 0)" >> $ENV_FILENAME

You can then set the .env_encoded file inside of your docker-compose.yml:

yaml
kestra:
  env_file: .env_encoded

Access Service Account inside of Kestra

You can now access this inside of Kestra with the following pebble expression:

yaml
"{{ secret('GCP_SERVICE_ACCOUNT') }}"

With this, we can add this to the serviceAccount property like so:

yaml
- id: upload
  type: io.kestra.plugin.googleworkspace.drive.Upload
  from: "{{ inputs.file }}"
  parents:
    - "1HuxzpLt1b0111MuKMgy8wAv-m9Myc1E_"
  name: "My awesome CSV"
  contentType: "text/csv"
  mimeType: "application/vnd.google-apps.spreadsheet"
  serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT') }}"
yaml
- id: fetch
  type: io.kestra.plugin.gcp.bigquery.Query
  fetch: true
  sql: |
    SELECT 1 as id, "John" as name
    UNION ALL
    SELECT 2 as id, "Doe" as name
  serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT') }}"

Set the Service Account with PluginDefaults

If you're using multiple tasks that will require the service account secret, you can set up a Plugin Default to apply this property to all tasks of this type. For example:

yaml
tasks:
  - id: "upload"
    type: "io.kestra.plugin.googleworkspace.drive.Upload"
    from: "{{ inputs.file }}"
    parents:
      - "1HuxzpLt1b0111MuKMgy8wAv-m9Myc1E_"
    name: "My awesome CSV"
    contentType: "text/csv"
    mimeType: "application/vnd.google-apps.spreadsheet"

pluginDefaults:
  - type: io.kestra.plugin.googleworkspace.drive.Upload
    values:
      serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT') }}"

Configuring Secrets in the Enterprise Edition

In Kestra Enterprise Edition, secrets can be managed directly from the UI meaning there's no need to encode them in base64. To learn more about this, check out the secrets page.

GOOGLE_APPLICATION_CREDENTIALS

While you can use the GOOGLE_APPLICATION_CREDENTIALS environment variable, this is not advised as you'll need to mount the JSON file to Docker which isn't always possible depending on how you've setup Kestra.

Google App Passwords

For some Google applications, like Gmail, you won't use a service account for authenticating. Instead, you'll use a normal username and password associated with a Google account. However, this doesn't work if your account has 2 factor authenication enabled. In this case, you'll need to generate an App Password. You can do this by going to Manage your Google Account, then go to Security. Select the App Passwords option and you'll be able to Generate a new one. This can be used where you'd put your normal password to connect it to Kestra.

Was this page helpful?