​Secret

Secret is a mechanism that allows you to securely store sensitive information, such as passwords and API keys, and retrieve them in your flows.

To retrieve secrets in a flow, use the secret() function, e.g. "{{ secret('API_TOKEN'') }}". You can leverage your existing secrets manager as a secrets backend.

Your flows often need to interact with external systems. To do that, they need to programmatically authenticate using passwords or API keys. Secrets help you securely store such variables and avoid hard-coding sensitive information within your workflow code.

You can leverage the secret() function, described in the function section, to retrieve sensitive variables within your flow code.

Secrets in the Open-Source version

When using the open-source version, sensitive variables can be managed using base64-encoded environment variables. The section below demonstrates several ways to encode those values and use them in your Kestra instance.

Manual encoding using a CLI command

Imagine that so far, you were setting the following environment variable:

export MYPASSWORD=password

Here is how you can encode the sensitive value of that environment variable:

echo -n "password" | base64

This should output the value: cGFzc3dvcmQ=.

To use that value as a Secret in your Kestra instance, you would need to add a prefix SECRET_ to the variable key (here: SECRET_MYPASSWORD) and set that key to the encoded value:

export SECRET_MYPASSWORD=cGFzc3dvcmQ=

If you would add the environment variable to the kestra container section in a Docker Compose file, it would look as follows:

  kestra:
    image: kestra/kestra:latest-full
    environment:
      SECRET_MYPASSWORD: cGFzc3dvcmQ=

This variable can then be used in a flow using the {{ secret('secret_key_name') }} syntax:

id: secretTest
namespace: dev
tasks:
  - id: hello
    type: io.kestra.core.tasks.log.Log
    message: "{{ secret('MYPASSWORD') }}"

When executing that flow, you should see the output password in your logs. Why password rather than the encoded value cGFzc3dvcmQ=? Because each secret is base64-decoded during a flow's execution. This means that only base64-encoded environment variables can be used with the secret() function. To reference not-e_ncoded environment variables, use the syntax {{envs.lowercase_environment_variable_key}} instead.

Make sure to add your custom environment variables before starting Kestra. Environment variables are loaded at the JVM startup, so adding new variables or modifying existing ones requires restarting your instance. To manage such changes without downtime, you can either try the Enterprise Edition or restart your server during a planned maintenance window.

Convert all variables in an .env file

The previous section showed the process for one Secret. But what if you have tens or hundreds of them? This is where .env file can come in handy.

Let's assume that you have an .env file with the following content:

MYPASSWORD=password
GITHUB_ACCESS_TOKEN=mypat
AWS_ACCESS_KEY_ID=myawsaccesskey
AWS_SECRET_ACCESS_KEY=myawssecretaccesskey

Make sure to keep the last line empty, otherwise the bash script below won't encode the last secret AWS_SECRET_ACCESS_KEY correctly.

Using the bash script shown below, you can:

1. Encode all values using base64-encoding
2. Add a SECRET_ prefix to all environment variable names
3. Store the result as .env_encoded

while IFS='=' read -r key value; do
    echo "SECRET_$key=$(echo -n "$value" | base64)";
done < .env > .env_encoded

The .env_encoded file should look as follows:

SECRET_MYPASSWORD=cGFzc3dvcmQ=
SECRET_GITHUB_ACCESS_TOKEN=bXlwYXQ=
SECRET_AWS_ACCESS_KEY_ID=bXlhd3NhY2Nlc3NrZXk=
SECRET_AWS_SECRET_ACCESS_KEY=bXlhd3NzZWNyZXRhY2Nlc3NrZXk=

Then, in your Docker Compose file, you can replace:

  kestra:
    image: kestra/kestra:latest-full
    env_file:
      - .env

with the encoded version of the file:

  kestra:
    image: kestra/kestra:latest-full
    env_file:
      - .env_encoded

After you've started Kestra using docker compose up -d, you should be able to run the flow shown below and see decoded secret values in the logs output:

id: secretTest
namespace: dev
tasks:
  - id: hello
    type: io.kestra.core.tasks.log.Log
    message: |
      {{ secret('MYPASSWORD') }}
      {{ secret('GITHUB_ACCESS_TOKEN') }}
      {{ secret('AWS_ACCESS_KEY_ID') }}
      {{ secret('AWS_SECRET_ACCESS_KEY') }}

Note that we didn't include the prefix SECRET_ when calling the secret('MYPASSWORD') function. The prefix is only required as a name of the environment variable to ensure that Kestra is only looking for the relevant environment variables without loading unnecessary system variables. This improves both performance and security as it ensures that Kestra only reads environment variables that are needed.

Use a macro within your .env file

As an alternative to replacing values in your environment variables by encoded counterparts, you may also leverage the base64encode macro and keep the values intact.

The original .env file:

MYPASSWORD=password
GITHUB_ACCESS_TOKEN=mypat
AWS_ACCESS_KEY_ID=myawsaccesskey
AWS_SECRET_ACCESS_KEY=myawssecretaccesskey

can be modified to the following format:

SECRET_MYPASSWORD={{ "password" | base64encode }}
SECRET_GITHUB_ACCESS_TOKEN={{ "mypat" | base64encode }}
SECRET_AWS_ACCESS_KEY_ID={{ "myawsaccesskey" | base64encode }}
SECRET_AWS_SECRET_ACCESS_KEY={{ "myawssecretaccesskey" | base64encode }}

Secrets in the Enterprise Edition

Adding a new Secret from the UI

If you are using a managed Kestra version, you can add new Secrets directly from the UI. In the left navigation menu, go to Namespaces, select the namespace to which you want to add a new secret. Then, add a new secret within the Secrets tab.

Here, we add a new secret with a key MY_SECRET:

Using Secrets in your flows

Here is a simple example showing how to retrieve that new secret in a flow:

id: secretTest
namespace: dev
tasks:
  - id: hello
    type: io.kestra.core.tasks.log.Log
    message: "{{ secret('MY_SECRET') }}"

When you execute that flow, you should see mysecretvalue in the logs output.

Secret Management backends

Kestra Enterprise Edition provides additional secret management backends and integrations with secrets managers. See the Secrets Manager page for more details.