Namespace Files

Manage Namespace Files and how to use them in your flows.

What are Namespace Files

Namespace Files are files tied to a given namespace. You can think of Namespace Files as equivalent of a project in your local IDE or a copy of your Git repository.

Namespace Files can hold Python files, R or Node.js scripts, SQL queries, dbt or Terraform projects, and many more.

You can synchronize your Git repository with a specific namespace to orchestrate dbt, Terraform or Ansible, or any other project that contains code and configuration files.

Once you add any file to a namespace, you can reference it inside of your flows using the read() function in EVERY task or trigger from the same namespace.

For instance, if you add a SQL query called my_query.sql to the queries directory in the company.team namespace, you can reference it in any Query task or any JDBC Trigger like so: {{ read('queries/my_query.sql') }}.

Here is an example showing how you can use the read() function in a ClickHouse Trigger to read a SQL query stored as a Namespace File:

yaml

id: jdbc_trigger
namespace: company.team

tasks:
  - id: for_each_row
    type: io.kestra.plugin.core.flow.ForEach
    values: "{{ trigger.rows }}"
    tasks:
      - id: return
        type: io.kestra.plugin.core.debug.Return
        format: "{{ json(taskrun.value) }}"

triggers:
  - id: query_trigger
    type: io.kestra.plugin.jdbc.clickhouse.Trigger
    interval: "PT5M"
    url: jdbc:clickhouse://127.0.0.1:56982/
    username: "{{ secret('CLICKHOUSE_USERNAME') }}"
    password: "{{ secret('CLICKHOUSE_PASSWORD') }}"
    sql: "{{ read('queries/my_query.sql') }}" # 🚀 The read() function reads the content of the file as a string!
    fetchType: FETCH

Note: we didn't have to use the namespaceFiles.enabled: true property — that property is only required to inject the entire directory of files from the namespace into the working directory of a script (e.g. a Python task). More on that in the subsequent sections of this page.

Why use Namespace Files

Namespace Files offer a simple way to organize your code and configuration files. Before Namespace Files, you had to store your code and configuration files in a Git repository and then clone that repository at runtime using the git.Clone task. With Namespace Files, you can store your code and configuration files directly in the Kestra's internal storage backend. That storage backend can be your local directory or an S3 bucket to ensure maximum security and privacy.

Namespace Files make it easy to:

orchestrate Python, R, Node.js, SQL, and more, without having to worry about code dependencies, packaging and deployments — simply add your code in the embedded Code Editor or sync your Git repository with a given namespace
manage your code for a given project or team in one place, even if those files are stored in different Git repositories, or even different Git providers
share your code and configuration files between workflows and team members in your organization
orchestrate complex projects that require the code to be separated into multiple scripts, queries or modules.

How to add Namespace Files

Embedded Code Editor

The easiest way to get started with Namespace Files is to use the embedded Code Editor. This allows you to easily add custom scripts, queries and configuration files along with your flow YAML configuration files.

Get started by selecting a namespace from the dropdown menu. If you type a name of a namespace that doesn't exist yet, Kestra will create it for you.

Then, add a new file, e.g., a Python script. Add a folder named scripts and a file called hello.py with the following content:

python

print("Hello from the Editor!")

Once you added a file, you can use it in your flow:

yaml

id: editor
namespace: company.team

tasks:
  - id: hello
    type: io.kestra.plugin.scripts.python.Commands
    namespaceFiles:
      enabled: true
    commands:
      - python scripts/hello.py

The Execute button allows you to run your flow directly from the Code Editor. Click on the Execute button to run your flow. You should then see the Execution being created in a new browser tab and once you navigate to the Logs tab, you should see a friendly message Hello from the Editor! in the logs.

PushNamespaceFiles and SyncNamespaceFiles Tasks

There's 2 tasks to help you automatically manage your namespace files with Git. This allows you to sync the latest changes from a Git repository.

This example will push Namespace Files you already have in Kestra to a Git repository for you:

yaml

id: push_to_git
namespace: system

tasks:
  - id: commit_and_push
    type: io.kestra.plugin.git.PushNamespaceFiles
    username: git_username
    password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
    url: https://github.com/git_username/scripts
    branch: dev
    namespace: company.team
    files:
      - "example.py"
    gitDirectory: _files
    commitMessage: "add namespace files"
    dryRun: true

This example will sync Namespace Files inside of a Git repository to your Kestra instance:

yaml

id: sync_files_from_git
namespace: system

tasks:
  - id: sync_files
    type: io.kestra.plugin.git.SyncNamespaceFiles
    username: git_username
    password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
    url: https://github.com/git_username/scripts
    branch: main
    namespace: git
    gitDirectory: _files
    dryRun: true

Check out the dedicated guides for more information:

GitHub Actions CI/CD

You can leverage our official GitHub Action called deploy-action to synchronize your Git repository with a given namespace. This is useful if you want to orchestrate complex Python modules, dbt projects, Terraform or Ansible infrastructure, or any other project that contains code and configuration files with potentially multiple nested directories and files.

Here is a simple example showing how you can deploy all scripts from the scripts directory in your Git branch to the prod namespace:

yaml

name: Kestra CI/CD
on:
  push:
    branches:
      - main
jobs:
  prod:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: deploy-scripts-to-prod
        uses: kestra-io/deploy-action@master
        with:
          resource: namespace_files
          namespace: prod
          directory: ./scripts # directory in the Git repository
          to: ./scripts # remote directory in the namespace
          server: https://demo.kestra.io/
          user: your_username
          password: ${{secrets.KESTRA_PASSWORD}}

When creating a service account role for the GitHub Action in the Enterprise Edition, you need to grant the FLOWS permission to the Role.

Terraform Provider

You can use the kestra_namespace_file resource from the official Kestra Terraform Provider to deploy all your custom script files from a specific directory to a given Kestra namespace.

Here is a simple example showing how you can synchronize an entire directory of scripts from the directory src with the company.team namespace using Terraform:

hcl

resource "kestra_namespace_file" "prod_scripts" {
  for_each  = fileset(path.module, "src/**")
  namespace = "company.team"
  filename   = each.value # or "/${each.value}"
  content   = file(each.value)
}

Deploy Namespace Files from Git via CLI

You can also use the Kestra CLI to deploy all your custom script files from a specific directory to a given Kestra namespace. Here is a simple example showing how you can synchronize an entire directory of local scripts with the prod namespace using the Kestra CLI:

bash

./kestra namespace files update prod /Users/anna/gh/KESTRA_REPOS/scripts --server=http://localhost:8080 --user=rick:password

In fact, you can even use that command directly in a flow. You can attach a schedule or a webhook trigger to automatically execute that flow anytime you push/merge changes to your Git repository, or on a regular schedule.

Here is an example of a flow that synchronizes an entire directory of local scripts with the prod namespace:

yaml

id: ci
namespace: company.team

variables:
  host: http://host.docker.internal:28080/

tasks:
  - id: deploy
    type: io.kestra.plugin.core.flow.WorkingDirectory
    tasks:
      - id: clone
        type: io.kestra.plugin.git.Clone
        url: https://github.com/kestra-io/scripts
        branch: main

      - id: deploy_files
        type: io.kestra.plugin.scripts.shell.Commands
        warningOnStdErr: false
        taskRunner:
          type: io.kestra.plugin.core.runner.Process
        commands:
          - /app/kestra namespace files update prod . . --server={{vars.host}}

Note that the two dots in the command /app/kestra namespace files update prod . . indicate that we want to sync an entire directory of files cloned from the Git repository to the root directory of the prod namespace. If you wanted to e.g. sync that repository to the scripts directory, you would use the following command: /app/kestra namespace files update prod . scripts. The syntax of that command follows the structure:

bash

/app/kestra namespace files update <namespace> <local_directory> <remote_directory>

To reproduce that flow, start Kestra using the following command:

bash

docker run --pull=always --rm -it -p 28080:8080  kestra/kestra:latest  server local

Then, open the Kestra UI at http://localhost:28080 and create a new flow with the content above. Once you execute the flow, you should see the entire directory from the scripts repository being synchronized with the prod namespace.

How to use Namespace Files in your flows

There are multiple ways to use Namespace Files in your flows. You can use the read() function to read the content of a file as a string, point to the file path in the supported tasks or use a dedicated task to retrieve it as an output.

Usually, pointing to a file location, rather than reading the file's content, is required when you want to use a file as an input to a CLI command, e.g. in a Commands task such as io.kestra.plugin.scripts.python.Commands or io.kestra.plugin.scripts.node.Commands. In all other cases, the read() function can be used to read the content of a file as a string e.g. in Query or Script tasks.

You can also use the io.kestra.plugin.core.flow.WorkingDirectory task to read namespace files there and then use them in child tasks that require reading the file path in CLI commands e.g. python scipts/hello.py.

The `read()` function

Note how the script in the first section used the read() function to read the content of the scripts/hello.py file as a string using the expression "{{ read('scripts/hello.py') }}". It'a important to remeber that this function reads the content of the file as a string. Therefore, you should use that function only in tasks that expect a string as an input, e.g., io.kestra.plugin.scripts.python.Script or io.kestra.plugin.scripts.node.Script, rather than io.kestra.plugin.scripts.python.Commands or io.kestra.plugin.scripts.node.Commands.

The read() function allows you to read the content of a Namespace File stored in the Kestra's internal storage backend. The read() function takes a single argument, which is the absolute path to the file you want to read. The path must point to a file stored in the same namespace as the flow you are executing.

In this example, we have a namespace file called example.txt that contains the text Hello, World!. We can print the content to the logs by using {{ read('example.txt') }}:

yaml

id: files
namespace: company.team

tasks:
  - id: log
    type: io.kestra.plugin.core.log.Log
    message: "{{ read('example.txt') }}"

`namespaceFiles.enabled` on supported tasks

With supported tasks, such as the io.kestra.plugin.scripts group, we can access files using their path and enabling the task to read namespace files.

Here is a simple weather.py script that reads a secret to talk to a Weather Data API:

python

import requests
api_key = '{{ secret("WEATHER_DATA_API_KEY") }}'
url = f"https://api.openweathermap.org/data/2.5/weather?q=Paris&APPID={api_key}"
weather_data = requests.get(url)
print(weather_data.json())

And here is the flow that uses the script:

yaml

id: weather_data
namespace: company.team

tasks:
  - id: get_weather
    type: io.kestra.plugin.scripts.python.Commands
    namespaceFiles:
      enabled: true
      include:
        - scripts/weather.py
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
    containerImage: ghcr.io/kestra-io/pydata:latest
    commands:
      - python scripts/weather.py

`namespaceFiles` property

The example above uses the include field to only allow the scripts/weather.py file to be accessible by the task.

We can control what namespace files are available to our flow with the namespaceFiles property.

namespaceFiles has 3 attributes:

enabled: when set to true enables all files in that namespace to be visible to the task
include: allows you to specify files you want to be accessible by the task
exclude: allows you to specify files you don't want to be accessible by the task

Namespace Tasks

You can use the Namespace Tasks to upload, download and delete tasks in Kestra.

In the example below, we have a namespace file called example.ion that we want to convert to a csv file. We can use the DownloadFiles task to generate an output that contains the file so we can easily pass it dynamically to the IonToCsv task.

yaml

id: files
namespace: company.team
tasks:
  - id: namespace
    type: io.kestra.plugin.core.namespace.DownloadFiles
    namespace: company.team
    files:
      - example.ion

  - id: ion_to_csv
    type: io.kestra.plugin.serdes.csv.IonToCsv
    from: "{{ outputs.namespace.files['/example.ion'] }}"

Include / Exclude Namespace Files

You can selectively include or exclude namespace files.

Say you have multiple namespace files present: file1.txt, file2.txt, file3.json, file4.yml. You can selectively include multiple files using include attribute under namespaceFiles as shown below:

yaml

id: include_namespace_files
namespace: company.team

tasks:
  - id: include_files
    type: io.kestra.plugin.scripts.shell.Commands
    namespaceFiles:
      enabled: true
      include:
        - file1.txt
        - file3.json
    commands:
      - ls

The include_files task will list all the included files, i.e. file1.txt and file3.json as only those got included from the namespace through include.

The exclude, on the other hand, includes all the namespace files except those specified under exclude.

yaml

id: exclude_namespace_files
namespace: company.team

tasks:
  - id: exclude_files
    type: io.kestra.plugin.scripts.shell.Commands
    namespaceFiles:
      enabled: true
      exclude:
        - file1.txt
        - file3.json
    commands:
      - ls

The exclude_files task from the above flow will list file2.txt and file4.yml, i.e. all the namespace files except those that were excluded using exclude.

Was this page helpful?

DocsConcepts

ConceptsRevision

​Namespace ​Files

Namespace Files