Namespace Files in Kestra – Manage Project Assets

Manage Namespace Files and how to use them in your flows.

Namespace Files – manage project assets

Namespace Files are files tied to a given namespace. You can think of Namespace Files as the equivalent of a project in your local IDE or a copy of your Git repository.

Namespace Files can hold Python files, R or Node.js scripts, SQL queries, dbt or Terraform projects, and much more.

You can synchronize your Git repository with a specific namespace to orchestrate dbt, Terraform or Ansible, or any other project that contains code and configuration files.

Once you add any file to a namespace, you can reference it inside your flows using the read() function in EVERY task or trigger from the same namespace.

For instance, if you add a SQL query called my_query.sql to the queries directory in the company.team namespace, you can reference it in any Query task or any JDBC Trigger like so: {{ read('queries/my_query.sql') }}.

Here is an example showing how you can use the read() function in a ClickHouse Trigger to read a SQL query stored as a Namespace File:

id: jdbc_trigger
namespace: company.team

tasks:
  - id: for_each_row
    type: io.kestra.plugin.core.flow.ForEach
    values: "{{ trigger.rows }}"
    tasks:
      - id: return
        type: io.kestra.plugin.core.debug.Return
        format: "{{ json(taskrun.value) }}"

triggers:
  - id: query_trigger
    type: io.kestra.plugin.jdbc.clickhouse.Trigger
    interval: "PT5M"
    url: jdbc:clickhouse://127.0.0.1:56982/
    username: "{{ secret('CLICKHOUSE_USERNAME') }}"
    password: "{{ secret('CLICKHOUSE_PASSWORD') }}"
    sql: "{{ read('queries/my_query.sql') }}" # 🚀 The read() function reads the content of the file as a string!
    fetchType: FETCH

Why use Namespace Files

Namespace Files offer a simple way to organize your code and configuration files. Before Namespace Files, you had to store your code and configuration files in a Git repository and then clone that repository at runtime using the git.Clone task. With Namespace Files, you can store your code and configuration files directly in the Kestra’s internal storage backend. That storage backend can be your local directory or an S3 bucket to ensure maximum security and privacy.

Namespace Files make it easy to:

• orchestrate Python, R, Node.js, SQL, and more without having to worry about code dependencies, packaging, and deployments — simply add your code in the embedded Code Editor or sync your Git repository with a given namespace
• manage your code for a given project or team in one place, even if those files are stored in different Git repositories or even different Git providers
• share your code and configuration files between workflows and team members in your organization
• orchestrate complex projects that require the code to be separated into multiple scripts, queries, or modules.

How to add Namespace Files

Embedded code editor

While creating or editing a Flow, you can access Namespace Files from the Files tab. You can easily write, import, or paste custom scripts, queries, and configuration files.

To start, add a new file, (e.g., a Python script). Add a folder named scripts and a file called hello.py with the following content:

print("Hello from the Editor!")

Once you added a file, you can use it in your flow:

id: editor
namespace: company.team

tasks:
  - id: hello
    type: io.kestra.plugin.scripts.python.Commands
    namespaceFiles:
      enabled: true
    commands:
      - python scripts/hello.py

The Execute button allows you to run your flow directly from the Code Editor. Click on the Execute button to run your flow. You then see the Execution running, and once you navigate to the Logs tab, you should see a friendly message Hello from the Editor! in the logs.

Namespace Files Revision History

Namespace Files include revision history just like flows, so you can inspect or roll back earlier uploads without leaving the Editor.

• First upload of a path is stored as queries/my_query.sql and treated as version 0 for backward compatibility.
• Each subsequent upload keeps queries/my_query.sql as the latest version while adding suffixed revisions such as queries/my_query.sql.v1, queries/my_query.sql.v2, and so on.
• Older revisions remain available under their suffixed filenames, letting you compare and restore as needed.

To access a file’s revision history, right-click on the file.

From the history, view, compare, and restore prior versions.

From the Revisions list, you can delete a given revision or all revisions older than the selected one. You will be prompted to confirm this choice, as there is no possible way to restore a revision once is has been deleted.

PushNamespaceFiles and SyncNamespaceFiles tasks

There are two tasks to help you automatically manage your namespace files with Git. This allows you to sync the latest changes from a Git repository.

This example pushes Namespace Files you already have in Kestra to a Git repository for you:

id: push_to_git
namespace: system

tasks:
  - id: commit_and_push
    type: io.kestra.plugin.git.PushNamespaceFiles
    username: git_username
    password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
    url: https://github.com/git_username/scripts
    branch: dev
    namespace: company.team
    files:
      - "example.py"
    gitDirectory: _files
    commitMessage: "add namespace files"
    dryRun: true

This example syncs Namespace Files inside of a Git repository to your Kestra instance:

id: sync_files_from_git
namespace: system

tasks:
  - id: sync_files
    type: io.kestra.plugin.git.SyncNamespaceFiles
    username: git_username
    password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
    url: https://github.com/git_username/scripts
    branch: main
    namespace: git
    gitDirectory: _files
    dryRun: true

Check out the dedicated guides for more information:

• PushNamespaceFiles
• SyncNamespaceFiles

GitHub Actions CI/CD

You can leverage our official GitHub Action called deploy-action to synchronize your Git repository with a given namespace. This is useful if you want to orchestrate complex Python modules, dbt projects, Terraform or Ansible infrastructure, or any other project that contains code and configuration files with potentially multiple nested directories and files.

Below is a simple example showing how you can deploy all scripts from the scripts directory in your Git branch to the prod namespace:

name: Kestra CI/CD
on:
  push:
    branches:
      - main
jobs:
  prod:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: deploy-scripts-to-prod
        uses: kestra-io/deploy-action@master
        with:
          resource: namespace_files
          namespace: prod
          directory: ./scripts # directory in the Git repository
          to: ./scripts # remote directory in the namespace
          server: https://demo.kestra.io/
          user: your_username
          password: ${{secrets.KESTRA_PASSWORD}}

Terraform provider

You can use the kestra_namespace_file resource from the official Kestra Terraform Provider to deploy all your custom script files from a specific directory to a given Kestra namespace.

Below is a simple example showing how you can synchronize an entire directory of scripts from the directory src with the company.team namespace using Terraform:

resource "kestra_namespace_file" "prod_scripts" {
  for_each  = fileset(path.module, "src/**")
  namespace = "company.team"
  filename   = each.value # or "/${each.value}"
  content   = file(each.value)
}

Deploy namespace files from Git via CLI

You can also use the Kestra CLI to deploy all your custom script files from a specific directory to a given Kestra namespace. Below is a simple example showing how you can synchronize an entire directory of local scripts with the prod namespace using the Kestra CLI:

./kestra namespace files update prod /Users/anna/gh/KESTRA_REPOS/scripts --server=http://localhost:8080 --user=rick:password

In fact, you can even use that command directly in a flow. You can attach a schedule or a webhook trigger to automatically execute that flow anytime you push/merge changes to your Git repository or on a regular schedule.

Below is an example of a flow that synchronizes an entire directory of local scripts with the prod namespace:

id: ci
namespace: company.team

variables:
  host: http://host.docker.internal:28080/

tasks:
  - id: deploy
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
      - id: clone
        type: io.kestra.plugin.git.Clone
        url: https://github.com/kestra-io/scripts
        branch: main

      - id: deploy_files
        type: io.kestra.plugin.scripts.shell.Commands
        taskRunner:
          type: io.kestra.plugin.core.runner.Process
        commands:
          - /app/kestra namespace files update prod . . --server={{vars.host}}

Note that the two dots in the command /app/kestra namespace files update prod . . indicate that we want to sync an entire directory of files cloned from the Git repository to the root directory of the prod namespace. If you wanted to sync that repository to the scripts directory, you would use the following command: /app/kestra namespace files update prod . scripts. The syntax of that command follows the structure:

/app/kestra namespace files update <namespace> <local_directory> <remote_directory>

To reproduce that flow, start Kestra using the following command:

docker run --pull=always --rm -it -p 28080:8080  kestra/kestra:latest  server local

Next, open the Kestra UI at http://localhost:28080 and create a new flow with the content above. Once you execute the flow, you then see the entire directory from the scripts repository being synchronized with the prod namespace.

How to use Namespace Files in your flows

There are multiple ways to use Namespace Files in your flows. You can use the read() function to read the content of a file as a string, point to the file path in the supported tasks, or use a dedicated task to retrieve it as an output.

Usually, pointing to a file location, rather than reading the file’s content, is required when you want to use a file as an input to a CLI command (e.g., in a Commands task such as io.kestra.plugin.scripts.python.Commands or io.kestra.plugin.scripts.node.Commands). In all other cases, the read() function can be used to read the content of a file as a string (e.g., in Query or Script tasks).

You can also use the io.kestra.plugin.core.flow.WorkingDirectory task to read namespace files there and then use them in child tasks that require reading the file path in CLI commands for example like: python scipts/hello.py.

The read() function

Note how the script in the first section used the read() function to read the content of the scripts/hello.py file as a string using the expression "{{ read('scripts/hello.py') }}". It’s important to remember that this function reads the content of the file as a string. Therefore, you should use that function only in tasks that expect a string as an input like io.kestra.plugin.scripts.python.Script or io.kestra.plugin.scripts.node.Script, rather than io.kestra.plugin.scripts.python.Commands or io.kestra.plugin.scripts.node.Commands.

The read() function allows you to read the content of a Namespace File stored in the Kestra’s internal storage backend. The read() function takes a single argument, which is the absolute path to the file you want to read. The path must point to a file stored in the same namespace as the flow you are executing.

In this example, we have a namespace file called example.txt that contains the text Hello, World!. We can print the content to the logs by using {{ read('example.txt') }}:

id: files
namespace: company.team

tasks:
  - id: log
    type: io.kestra.plugin.core.log.Log
    message: "{{ read('example.txt') }}"

namespaceFiles.enabled on supported tasks

With supported tasks, such as the io.kestra.plugin.scripts group, we can access files using their path and enabling the task to read namespace files.

Below is a simple weather.py script that reads a secret to talk to a Weather Data API:

import requests
api_key = '{{ secret("WEATHER_DATA_API_KEY") }}'
url = f"https://api.openweathermap.org/data/2.5/weather?q=Paris&APPID={api_key}"
weather_data = requests.get(url)
print(weather_data.json())

Next, is a flow that uses the script:

id: weather_data
namespace: company.team

tasks:
  - id: get_weather
    type: io.kestra.plugin.scripts.python.Commands
    namespaceFiles:
      enabled: true
      include:
        - scripts/weather.py
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
    containerImage: ghcr.io/kestra-io/pydata:latest
    commands:
      - python scripts/weather.py

namespaceFiles property

The example above uses the include field to only allow the scripts/weather.py file to be accessible by the task.

We can control what namespace files are available to our flow with the namespaceFiles property.

namespaceFiles has several configurable attributes:

• enabled: when set to true enables all files in that namespace to be visible to the task
• include: specifies files you want to be accessible by the task
• exclude: specifies files you don’t want to be accessible by the task
• namespaces: specifies a list of namespaces to search for files.
• ifExists: specifies what to do in the instance a Namespace file already exists in the working directory
• folderPerNamespace: a boolean property that mounts namespace files in separate directories (set to false by default) rather than all files to the root of the working directory

The namespaces attribute can be used like in the following example:

id: namespace_files_example
namespace: dev.test

tasks:

  - id: namespace
    type: io.kestra.plugin.scripts.python.Commands
    namespaceFiles:
      enabled: true
      namespaces:
        - "dev.test"
        - "company"
    commands:
      - python test.py


  - id: namespace2
    type: io.kestra.plugin.scripts.python.Script
    namespaceFiles:
      enabled: true
    script: "{{ read('test.py') }}"

The files are loaded in the namespace order, and only the latest version of a file is kept. Meaning if a file is present in the first and second namespace, only the file present on the second namespace will be loaded. In the first task, the test.py file from the company namespace will be used because priority is given from top to bottom of the list of namespaces. In the case of multiple files of the same name, the last listed namespace holds priority.

For the second task, the test.py file in the dev.test namespace will be used because no namespace has been defined in the read() function. If you want to fetch the test.py script from a different namespace, you need to explicitly define it as follows: "{{ read('test.py', namespace='company.team') }}".

The ifExists attribute has four possible options for behavior when tasks invoke a Namespace file that already exists in the working directory:

• OVERWRITE: set by default, adds a debug log that the file was overwritten
• FAIL: logs and ERROR and fails the task
• WARN: logs a WARNING but continues running the execution
• IGNORE: doesn’t overwrite the file or log any warnings

For example, in the following task the second instance of sample_python.py will overwrite the first:

id: test_workdir_issue
namespace: prod

tasks:
  - id: git_wdir
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
      - id: clone
        type: io.kestra.plugin.git.Clone
        branch: main
        url: https://github.com/kestra-io/examples

      - id: python_command_1
        type: io.kestra.plugin.scripts.python.Commands
        namespaceFiles:
          enabled: true
        commands:
          - python scripts/sample_python.py

      - id: python_command_2
        type: io.kestra.plugin.scripts.python.Commands
        namespaceFiles:
          enabled: true
          ifExists: OVERWRITE
        commands:
          - python scripts/sample_python.py

Namespace tasks

You can use the Namespace Tasks to upload, download, and delete tasks in Kestra.

In the example below, we have a namespace file called example.ion that we want to convert to a .csv file. We can use the DownloadFiles task to generate an output that contains the file so we can easily pass it dynamically to the IonToCsv task.

id: files
namespace: company.team
tasks:
  - id: namespace
    type: io.kestra.plugin.core.namespace.DownloadFiles
    namespace: company.team
    files:
      - example.ion

  - id: ion_to_csv
    type: io.kestra.plugin.serdes.csv.IonToCsv
    from: "{{ outputs.namespace.files['/example.ion'] }}"

Read more about the tasks below:

• UploadFiles
• DownloadFiles
• DeleteFiles

Include / exclude namespace files

You can selectively include or exclude namespace files.

Let’s say that you have multiple namespace files present: file1.txt, file2.txt, file3.json, file4.yml. You can selectively include multiple files using the include attribute under namespaceFiles as shown below:

id: include_namespace_files
namespace: company.team

tasks:
  - id: include_files
    type: io.kestra.plugin.scripts.shell.Commands
    namespaceFiles:
      enabled: true
      include:
        - file1.txt
        - file3.json
    commands:
      - ls

The include_files task lists all the included files. In the example above, these are file1.txt and file3.json as only those were included from the namespace through include.

The exclude attribute, alternatively, includes all the namespace files except those specified under exclude.

id: exclude_namespace_files
namespace: company.team

tasks:
  - id: exclude_files
    type: io.kestra.plugin.scripts.shell.Commands
    namespaceFiles:
      enabled: true
      exclude:
        - file1.txt
        - file3.json
    commands:
      - ls

The exclude_files task from the above flow lists file2.txt and file4.yml, all the namespace files except those that were excluded using exclude.

Pattern matching rules for include / exclude

• Patterns that do not start with / are automatically prefixed with **/, so they match recursively (e.g., file1.txt becomes **/file1.txt).
• Patterns that start with / match from the namespace root only (e.g., /config/settings.json).
• You can force explicit types with glob: or regex::
  • glob:/src/**/*.py
  • regex:^src/.*\\.py$

Examples:

namespaceFiles:
  enabled: true
  include:
    # Root-only matches
    - /file1.txt
    - /config/settings.json

    # Recursive matches (auto **/ prefix)
    - file1.txt          # becomes **/file1.txt
    - src/**             # becomes **/src/**

    # Explicit glob
    - glob:/src/**/*.py
    - glob:config/*.json

    # Regex
    - regex:^src/.*\.py$
    - regex:.*test.*\.json