​Namespace ​Management

How to govern secrets, variables, and plugin defaults on a namespace level.

Overview

Kestra is a multi-tenant platform. Each tenant can have multiple namespaces, and each namespace provides additional isolation and security.

Namespaces provide:

• a logical isolation of resources on top of the instance- or tenant-level isolation
• fine-grained access-control to secrets, variables, and task configuration.

Namespaces are particularly useful in environments with many users, teams, projects, and applications.

Namespace Management Benefits

Even though namespace is a required property of each flow, namespaces are not created by default. To illustrate this, let's look at the following flow:

id: hello_world
namespace: company.marketing
tasks:
  - id: log_task
    type: io.kestra.plugin.core.log.Log
    message: hi from {{ flow.namespace }}

This flow is assigned to the company.marketing namespace. However, if you navigate to the Namespaces page in the UI, you'll notice that the namespace itself doesn't exist yet. The company.marketing namespace is greyed out because it's just a placeholder:

You can only filter for existing namespaces. Once you are ready to turn a placeholder namespace into a fully-fledged namespace, you create it in one click:

Namespace-level features

The Namespace page allows you to configure secrets, plugin defaults, and variables that can be used within any flow in that namespace.

It allows your organization to centrally manage your secrets, variables, and task configuration while providing fine-grained access-control to those resources.

Since Kestra supports everything as code and from the UI, you can manage namespaces from the UI or programmatically (e.g., via our Terraform provider).

Secrets

On the Namespaces page, select the namespace where you want to define the secrets and go to the Secrets tab. Here, you will see all existing secrets associated with this namespace. Click on Add a secret button on the top right corner of the page.

You can now define the secret by providing the Key and the Secret value. Save the secret by clicking on the Save button at the bottom.

The secret key should now start appearing on the Secrets tab. You can edit the secret's value or delete the secret by clicking on the appropriate button towards the right of the secret row. You can reference the secret in the flow by using the key, for example, "{{ secret('MYSQL_PASSWORD') }}".

Here is how you can use it in a flow:

id: query-mysql
namespace: company.team

tasks:
  id: query
  type: "io.kestra.plugin.jdbc.mysql.Query"
  url: jdbc:mysql://localhost:3306/test
  username: root
  password: "{{ secret('MYSQL_PASSWORD') }}"
  sql: select * from employees
  fetchOne: true

Plugin Defaults

Plugin Defaults can also be defined at the namespace level. These plugin defaults are then applied for all tasks of the corresponding type defined in the flows under the same namespace.

On the Namespaces page, select the namespace where you want to define the plugin defaults and navigate to the Plugin defaults tab. You can add the plugin defaults here and save the changes by clicking on the Save button at the bottom of the page.

You can reference secrets and variables defined with the same namespace in the plugin defaults.

In the example below, you no longer need to add the password property for the MySQL query task as it's defined in your namespace-level pluginDefaults:

id: query-mysql
namespace: company.team

tasks:
  id: query
  type: "io.kestra.plugin.jdbc.mysql.Query"
  url: jdbc:mysql://localhost:3306/test
  username: root
  sql: select * from employees
  fetchOne: true

Variables

Variables defined at the namespace level can be used in any flow defined under the same namespace using the syntax: {{ namespace.variable_name }}.

On the Namespaces page, select the namespace where you want to define the variables. Go to the Variables tab. You can now define the variables on this page. Save the changes by clicking the Save button at the bottom of the page.

Here is an example flow where the namespace variable is used:

id: query-mysql
namespace: company.team

tasks:
  id: query
  type: "io.kestra.plugin.jdbc.mysql.Query"
  url: jdbc:mysql://localhost:3306/test
  username: "{{ namespace.mysql_user }}"
  sql: select * from employees
  fetchOne: true

Creating Namespaces

From the UI

The video below shows how you can create a namespace from the Kestra UI. After creating a namespace, we're adding:

• several new secrets
• a nested namespace variable that references one of these secrets
• a list of plugin defaults helping to use those pre-configured secrets and variables in all the tasks from the AWS and Git plugins.

From Terraform

Let's reproduce everything from the above video using Kestra's Terraform provider so that you know how to perform the same steps both from the UI and programmatically.

To create a namespace from Terraform, use the kestra_namespace resource.

First, configure your Terraform backend and add Kestra as a required provider:

terraform {
  backend "s3" {
    bucket = "kestraio"
    key    = "terraform.tfstate"
    region = "us-east-1"
  }
  required_providers {
    kestra = {
      source  = "kestra-io/kestra"
      version = "~>0.14"
    }
  }
}

provider "kestra" {
  url       = var.kestra_host
  username  = var.kestra_user
  password  = var.kestra_password
  tenant_id = var.kestra_tenant_id # only if you are using multi-tenancy
}

You can add a file main.tf to your Terraform project with the following content:

resource "kestra_namespace" "marketing" {
  namespace_id  = "marketing"
  description   = "Namespace for the marketing team"
}

The only required property is the namespace_id, which is the name of the namespace. The description and all other properties are optional.

Adding Variables and Plugin Defaults to a Namespace Terraform resource

You can add variables and plugin defaults directly to the namespace resource by pointing to the YAML configuration files.

First, create the variables_marketing.yml file:

github:
  token: "{{ secret('GITHUB_TOKEN') }}"

Then, create another file for task_defaults_marketing.yml:

- type: io.kestra.plugin.aws
  values:
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    region: us-east-1
    secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
- type: io.kestra.plugin.git
  values:
    password: "{{ render(namespace.github.token) }}"
    username: your-github-username

Finally, reference those files in your namespace resource definition:

resource "kestra_namespace" "marketing" {
  namespace_id  = "marketing"
  description   = "Namespace for the marketing team"
  variables     = file("variables_marketing.yml")
  task_defaults = file("task_defaults_marketing.yml")
}

Adding Secrets to a Namespace using Terraform

To programmatically add secrets to your namespace via Terraform, you can use the kestra_namespace_secret resource. Here is an example of adding multiple secrets to the marketing namespace:

resource "kestra_namespace_secret" "github_token" {
  namespace    = "marketing"
  secret_key   = "GITHUB_TOKEN"
  secret_value = var.github_token
}

resource "kestra_namespace_secret" "aws_access_key_id" {
  namespace    = "marketing"
  secret_key   = "AWS_ACCESS_KEY_ID"
  secret_value = var.aws_access_key_id
}

resource "kestra_namespace_secret" "aws_secret_access_key" {
  namespace    = "marketing"
  secret_key   = "AWS_SECRET_ACCESS_KEY"
  secret_value = var.aws_secret_access_key
}

Before referencing variables in your Terraform configuration, make sure to define them in your variables.tf file:

variable "github_token" {
  type = string
  sensitive = true
}

variable "aws_access_key_id" {
  type = string
  sensitive = true
}

variable "aws_secret_access_key" {
  type = string
  sensitive = true
}

variable "kestra_user" {
  type      = string
  sensitive = true
}

variable "kestra_password" {
  type      = string
  sensitive = true
}

variable "kestra_host" {
  type      = string
  sensitive = false
  default   = "https://us.kestra.cloud"
}

variable "kestra_tenant_id" {
  type      = string
  sensitive = false
  default   = "kestra-tech"
}

And add your secrets to the terraform.tfvars file:

github_token = "your-github-token"
aws_access_key_id = "your-aws-access-key-id"
aws_secret_access_key = "your-aws-secret-access-key"
kestra_user = "your-kestra-user"
kestra_password = "your-kestra-password"

Allowed Namespaces

As of Kestra 0.17, you can manage cross-namespace permissions in the Enterprise Edition.

When you navigate to any Namespace and go to the Edit tab, you can explicitly configure which namespaces are allowed to access flows and other resources related to that namespace. By default, all namespaces are allowed:

However, you can restrict that access if you want only specific namespaces (or no namespace at all) to trigger its corresponding resources.