Store sensitive information securely.
Secret is a mechanism that allows you to securely store sensitive information, such as passwords and API keys, and retrieve them in your flows.
To retrieve secrets in a flow, use the secret()
function, e.g. "{{ secret('API_TOKEN'') }}"
. You can leverage your existing secrets manager as a secrets backend.
Your flows often need to interact with external systems. To do that, they need to programmatically authenticate using passwords or API keys. Secrets help you securely store such variables and avoid hard-coding sensitive information within your workflow code.
You can leverage the secret()
function to retrieve sensitive variables within your flow code.
Secrets in the Enterprise Edition
Adding a new Secret from the UI
If you are using a managed Kestra version, you can add new Secrets directly from the UI. In the left navigation menu, go to Namespaces, select the namespace to which you want to add a new secret. Then, add a new secret within the Secrets tab.
Here, we add a new secret with a key MY_SECRET
:
Using Secrets in your flows
For a concrete example of using secrets in flows, check out our dedicated How-To Guide on Secrets.
Secret Management backends
Kestra Enterprise Edition provides additional secret management backends and integrations with secrets managers. See the Secrets Manager page for more details.
Secrets in the Open-Source version
When using the open-source version, sensitive variables can be managed using base64-encoded environment variables. The section below demonstrates several ways to encode those values and use them in your Kestra instance.
Manual encoding using a CLI command
Imagine that so far, you were setting the following environment variable:
export MYPASSWORD=myPrivateCode
Here is how you can encode the sensitive value of that environment variable:
echo -n "myPrivateCode" | base64
This should output the value: bXlQcml2YXRlQ29kZQ==
To use that value as a Secret in your Kestra instance, you would need to add a prefix SECRET_
to the variable key (here: SECRET_MYPASSWORD
) and set that key to the encoded value:
export SECRET_MYPASSWORD=bXlQcml2YXRlQ29kZQ==
If you would add the environment variable to the kestra
container section in a Docker Compose file, it would look as follows:
kestra:
image: kestra/kestra:latest
environment:
SECRET_MYPASSWORD: bXlQcml2YXRlQ29kZQ==
This secret can then be used in a flow using the {{ secret('MYPASSWORD') }}
syntax, and it will base64-decoded during flow execution. Make sure to not include the prefix SECRET_
when calling the secret('MYPASSWORD')
function, as this prefix is only there in the environment variable definition to prevent Kestra from treating other system variables as secrets (for better performance and increased security).
Lastly, shall you wish to reference any non_encoded environment variables in your flows definition, you can always use the syntax {{envs.lowercase_environment_variable_key}}
.
Note that Kestra has built-in protection to prevent its logs from revealing any encoded secret you would have defined.
Convert all variables in an .env
file
The previous section showed the process for one Secret. But what if you have tens or hundreds of them? This is where .env
file can come in handy.
Let's assume that you have an .env
file with the following content:
MYPASSWORD=password
GITHUB_ACCESS_TOKEN=mypat
AWS_ACCESS_KEY_ID=myawsaccesskey
AWS_SECRET_ACCESS_KEY=myawssecretaccesskey
Make sure to keep the last line empty, otherwise the bash script below won't encode the last secret AWS_SECRET_ACCESS_KEY correctly.
Using the bash script shown below, you can:
- Encode all values using base64-encoding
- Add a
SECRET_
prefix to all environment variable names - Store the result as
.env_encoded
while IFS='=' read -r key value; do
echo "SECRET_$key=$(echo -n "$value" | base64)";
done < .env > .env_encoded
The .env_encoded
file should look as follows:
SECRET_MYPASSWORD=cGFzc3dvcmQ=
SECRET_GITHUB_ACCESS_TOKEN=bXlwYXQ=
SECRET_AWS_ACCESS_KEY_ID=bXlhd3NhY2Nlc3NrZXk=
SECRET_AWS_SECRET_ACCESS_KEY=bXlhd3NzZWNyZXRhY2Nlc3NrZXk=
Then, in your Docker Compose file, you can replace:
kestra:
image: kestra/kestra:latest
env_file:
- .env
with the encoded version of the file:
kestra:
image: kestra/kestra:latest
env_file:
- .env_encoded
Use a macro within your .env
file
As an alternative to replacing values in your environment variables by encoded counterparts, you may also leverage the base64encode
macro and keep the values intact.
The original .env
file:
MYPASSWORD=password
GITHUB_ACCESS_TOKEN=mypat
AWS_ACCESS_KEY_ID=myawsaccesskey
AWS_SECRET_ACCESS_KEY=myawssecretaccesskey
can be modified to the following format:
SECRET_MYPASSWORD={{ "password" | base64encode }}
SECRET_GITHUB_ACCESS_TOKEN={{ "mypat" | base64encode }}
SECRET_AWS_ACCESS_KEY_ID={{ "myawsaccesskey" | base64encode }}
SECRET_AWS_SECRET_ACCESS_KEY={{ "myawssecretaccesskey" | base64encode }}
Was this page helpful?