Store sensitive information securely.
Secret is a mechanism that allows you to securely store sensitive information, such as passwords and API keys, and retrieve them in your flows.
To retrieve secrets in a flow, use the secret()
function, e.g. "{{ secret('API_TOKEN'') }}"
. You can leverage your existing secrets manager as a secrets backend.
Your flows often need to interact with external systems. To do that, they need to programmatically authenticate using passwords or API keys. Secrets help you securely store such variables and avoid hard-coding sensitive information within your workflow code.
You can leverage the secret()
function, described in the function section, to retrieve sensitive variables within your flow code.
Secrets in the Enterprise Edition
Adding a new Secret from the UI
If you are using a managed Kestra version, you can add new Secrets directly from the UI. In the left navigation menu, go to Namespaces, select the namespace to which you want to add a new secret. Then, add a new secret within the Secrets tab.
Here, we add a new secret with a key MY_SECRET
:
Using Secrets in your flows
Here is a simple example showing how to retrieve that new secret in a flow:
id: secretTest
namespace: company.team
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: "{{ secret('MY_SECRET') }}"
When you execute that flow, you should see mysecretvalue
in the logs output.
The purpose of this example is to show how to use secrets in your flows. In practice, you should avoid logging sensitive information.
Secret Management backends
Kestra Enterprise Edition provides additional secret management backends and integrations with secrets managers. See the Secrets Manager page for more details.
Secrets in the Open-Source version
When using the open-source version, sensitive variables can be managed using base64-encoded environment variables. The section below demonstrates several ways to encode those values and use them in your Kestra instance.
Manual encoding using a CLI command
Imagine that so far, you were setting the following environment variable:
export MYPASSWORD=password
Here is how you can encode the sensitive value of that environment variable:
echo -n "password" | base64
This should output the value: cGFzc3dvcmQ=
.
To use that value as a Secret in your Kestra instance, you would need to add a prefix SECRET_
to the variable key (here: SECRET_MYPASSWORD
) and set that key to the encoded value:
export SECRET_MYPASSWORD=cGFzc3dvcmQ=
If you would add the environment variable to the kestra
container section in a Docker Compose file, it would look as follows:
kestra:
image: kestra/kestra:latest
environment:
SECRET_MYPASSWORD: cGFzc3dvcmQ=
This variable can then be used in a flow using the {{ secret('secret_key_name') }}
syntax:
id: secretTest
namespace: company.team
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: "{{ secret('MYPASSWORD') }}"
When executing that flow, you should see the output password
in your logs. Why password
rather than the encoded value cGFzc3dvcmQ=
? Because each secret is base64-decoded during a flow's execution. This means that only base64-encoded environment variables can be used with the secret()
function. To reference not-e_ncoded environment variables, use the syntax {{envs.lowercase_environment_variable_key}}
instead.
Make sure to add your custom environment variables before starting Kestra. Environment variables are loaded at the JVM startup, so adding new variables or modifying existing ones requires restarting your instance. To manage such changes without downtime, you can either try the Enterprise Edition or restart your server during a planned maintenance window.
Convert all variables in an .env
file
The previous section showed the process for one Secret. But what if you have tens or hundreds of them? This is where .env
file can come in handy.
Let's assume that you have an .env
file with the following content:
MYPASSWORD=password
GITHUB_ACCESS_TOKEN=mypat
AWS_ACCESS_KEY_ID=myawsaccesskey
AWS_SECRET_ACCESS_KEY=myawssecretaccesskey
Make sure to keep the last line empty, otherwise the bash script below won't encode the last secret AWS_SECRET_ACCESS_KEY correctly.
Using the bash script shown below, you can:
- Encode all values using base64-encoding
- Add a
SECRET_
prefix to all environment variable names - Store the result as
.env_encoded
while IFS='=' read -r key value; do
echo "SECRET_$key=$(echo -n "$value" | base64)";
done < .env > .env_encoded
The .env_encoded
file should look as follows:
SECRET_MYPASSWORD=cGFzc3dvcmQ=
SECRET_GITHUB_ACCESS_TOKEN=bXlwYXQ=
SECRET_AWS_ACCESS_KEY_ID=bXlhd3NhY2Nlc3NrZXk=
SECRET_AWS_SECRET_ACCESS_KEY=bXlhd3NzZWNyZXRhY2Nlc3NrZXk=
Then, in your Docker Compose file, you can replace:
kestra:
image: kestra/kestra:latest
env_file:
- .env
with the encoded version of the file:
kestra:
image: kestra/kestra:latest
env_file:
- .env_encoded
After you've started Kestra using docker compose up -d
, you should be able to run the flow shown below and see decoded secret values in the logs output:
id: secretTest
namespace: company.team
tasks:
- id: hello
type: io.kestra.plugin.core.log.Log
message: |
{{ secret('MYPASSWORD') }}
{{ secret('GITHUB_ACCESS_TOKEN') }}
{{ secret('AWS_ACCESS_KEY_ID') }}
{{ secret('AWS_SECRET_ACCESS_KEY') }}
Note that we didn't include the prefix
SECRET_
when calling thesecret('MYPASSWORD')
function. The prefix is only required as a name of the environment variable to ensure that Kestra is only looking for the relevant environment variables without loading unnecessary system variables. This improves both performance and security as it ensures that Kestra only reads environment variables that are needed.
Use a macro within your .env
file
As an alternative to replacing values in your environment variables by encoded counterparts, you may also leverage the base64encode
macro and keep the values intact.
The original .env
file:
MYPASSWORD=password
GITHUB_ACCESS_TOKEN=mypat
AWS_ACCESS_KEY_ID=myawsaccesskey
AWS_SECRET_ACCESS_KEY=myawssecretaccesskey
can be modified to the following format:
SECRET_MYPASSWORD={{ "password" | base64encode }}
SECRET_GITHUB_ACCESS_TOKEN={{ "mypat" | base64encode }}
SECRET_AWS_ACCESS_KEY_ID={{ "myawsaccesskey" | base64encode }}
SECRET_AWS_SECRET_ACCESS_KEY={{ "myawssecretaccesskey" | base64encode }}
Was this page helpful?