​Secret

Store sensitive information securely.

Secret is a mechanism that allows you to securely store sensitive information, such as passwords and API keys, and retrieve them in your flows.

To retrieve secrets in a flow, use the secret() function, e.g. "{{ secret('API_TOKEN'') }}". You can leverage your existing secrets manager as a secrets backend.

Your flows often need to interact with external systems. To do that, they need to programmatically authenticate using passwords or API keys. Secrets help you securely store such variables and avoid hard-coding sensitive information within your workflow code.

You can leverage the secret() function to retrieve sensitive variables within your flow code.

Secrets in the Enterprise Edition

Adding a new Secret from the UI

If you are using a managed Kestra version, you can add new Secrets directly from the UI. In the left navigation menu, go to Namespaces, select the namespace to which you want to add a new secret. Then, add a new secret within the Secrets tab.

Here, we add a new secret with a key MY_SECRET:

Using Secrets in your flows

For a concrete example of using secrets in flows, check out our dedicated How-To Guide on Secrets.

Secret Management backends

Kestra Enterprise Edition provides additional secret management backends and integrations with secrets managers. See the Secrets Manager page for more details.

Secrets in the Open-Source version

When using the open-source version, sensitive variables can be managed using base64-encoded environment variables. The section below demonstrates several ways to encode those values and use them in your Kestra instance.

Manual encoding using a CLI command

Imagine that so far, you were setting the following environment variable:

export MYPASSWORD=myPrivateCode

Here is how you can encode the sensitive value of that environment variable:

echo -n "myPrivateCode" | base64

This should output the value: bXlQcml2YXRlQ29kZQ==

To use that value as a Secret in your Kestra instance, you would need to add a prefix SECRET_ to the variable key (here: SECRET_MYPASSWORD) and set that key to the encoded value:

export SECRET_MYPASSWORD=bXlQcml2YXRlQ29kZQ==

If you would add the environment variable to the kestra container section in a Docker Compose file, it would look as follows:

  kestra:
    image: kestra/kestra:latest
    environment:
      SECRET_MYPASSWORD: bXlQcml2YXRlQ29kZQ==

This secret can then be used in a flow using the {{ secret('MYPASSWORD') }} syntax, and it will base64-decoded during flow execution. Make sure to not include the prefix SECRET_ when calling the secret('MYPASSWORD') function, as this prefix is only there in the environment variable definition to prevent Kestra from treating other system variables as secrets (for better performance and increased security).

Lastly, shall you wish to reference any non_encoded environment variables in your flows definition, you can always use the syntax {{envs.lowercase_environment_variable_key}}.

Convert all variables in an .env file

The previous section showed the process for one Secret. But what if you have tens or hundreds of them? This is where .env file can come in handy.

Let's assume that you have an .env file with the following content:

MYPASSWORD=password
GITHUB_ACCESS_TOKEN=mypat
AWS_ACCESS_KEY_ID=myawsaccesskey
AWS_SECRET_ACCESS_KEY=myawssecretaccesskey

Make sure to keep the last line empty, otherwise the bash script below won't encode the last secret AWS_SECRET_ACCESS_KEY correctly.

Using the bash script shown below, you can:

1. Encode all values using base64-encoding
2. Add a SECRET_ prefix to all environment variable names
3. Store the result as .env_encoded

while IFS='=' read -r key value; do
    echo "SECRET_$key=$(echo -n "$value" | base64)";
done < .env > .env_encoded

The .env_encoded file should look as follows:

SECRET_MYPASSWORD=cGFzc3dvcmQ=
SECRET_GITHUB_ACCESS_TOKEN=bXlwYXQ=
SECRET_AWS_ACCESS_KEY_ID=bXlhd3NhY2Nlc3NrZXk=
SECRET_AWS_SECRET_ACCESS_KEY=bXlhd3NzZWNyZXRhY2Nlc3NrZXk=

Then, in your Docker Compose file, you can replace:

  kestra:
    image: kestra/kestra:latest
    env_file:
      - .env

with the encoded version of the file:

  kestra:
    image: kestra/kestra:latest
    env_file:
      - .env_encoded

Use a macro within your .env file

As an alternative to replacing values in your environment variables by encoded counterparts, you may also leverage the base64encode macro and keep the values intact.

The original .env file:

MYPASSWORD=password
GITHUB_ACCESS_TOKEN=mypat
AWS_ACCESS_KEY_ID=myawsaccesskey
AWS_SECRET_ACCESS_KEY=myawssecretaccesskey

can be modified to the following format:

SECRET_MYPASSWORD={{ "password" | base64encode }}
SECRET_GITHUB_ACCESS_TOKEN={{ "mypat" | base64encode }}
SECRET_AWS_ACCESS_KEY_ID={{ "myawsaccesskey" | base64encode }}
SECRET_AWS_SECRET_ACCESS_KEY={{ "myawssecretaccesskey" | base64encode }}