Sync Flows and Namespace Files from Git into Kestra

For the complete documentation index, see llms.txt. For a full content snapshot, see llms-full.txt. Append .md to any kestra.io/docs/* URL for plain Markdown.

Apply flows and namespace files from a Git repository to Kestra, with Git as the source of truth.

The SyncFlows and SyncNamespaceFiles tasks read a Git branch and create or update the matching flows and namespace files in Kestra. A changed resource in Git overwrites the version in Kestra, including edits made in the UI. To commit resources from Kestra to Git instead, see Push to Git.

Before you begin

  • A Git repository with your flows or namespace files, and a personal access token (PAT) with read access to it. Store the PAT as a secret, for example GITHUB_ACCESS_TOKEN.
  • Credentials for the Kestra API. Both tasks create and update resources through the API, so on an instance with authentication they need either task-level auth (and kestraUrl if the API isn’t reachable at the default URL) or default credentials configured with kestra.tasks.sdk.authentication.*, or at the namespace or tenant level in Enterprise Edition. See SDK authentication.

Sync flows

This flow syncs the flows in the flows directory of the main branch into the company.team namespace. With dryRun: true, it computes the changes without applying them:

id: sync_flows_from_git
namespace: company.ops
tasks:
- id: sync_flows
type: io.kestra.plugin.git.SyncFlows
url: https://github.com/my-org/kestra-flows
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
branch: main
targetNamespace: company.team
gitDirectory: flows
dryRun: true

Run the flow and open the task’s Outputs tab. The flows output is a file with one row per flow that the sync adds, updates, or deletes. Check it, then set dryRun: false and run the flow again to apply the changes.

  • targetNamespace is required. Every synced flow is written to this namespace, whatever namespace its YAML file declares.
  • Flows that fail validation stop the sync. Set ignoreInvalidFlows: true to log a warning and skip them instead.
  • By default, delete is false, so flows that exist in Kestra but not in Git are kept. Set delete: true to remove them, and run a dry run first to see what would be deleted.

Include child namespaces

Set includeChildNamespaces: true to sync subfolders of gitDirectory into child namespaces of targetNamespace. With targetNamespace: company.team, a flow at flows/tutorial/hello_world_1.yml is synced into company.team.tutorial. This matches the layout that PushFlows creates.

Sync namespace files

SyncNamespaceFiles works the same way for namespace files. This flow syncs the _files directory into the company.team namespace:

id: sync_files_from_git
namespace: company.ops
tasks:
- id: sync_files
type: io.kestra.plugin.git.SyncNamespaceFiles
url: https://github.com/my-org/kestra-flows
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
branch: main
namespace: company.team
gitDirectory: _files
dryRun: true
  • The dry-run diff is in the task’s files output.
  • By default, delete is false. Set delete: true to remove namespace files that aren’t in Git.
  • With includeChildNamespaces: true, a folder under gitDirectory named after an existing child namespace, such as _files/company.team.tutorial/, is synced into that namespace. This matches the layout that PushNamespaceFiles creates.

Sync one folder of a namespace

Use namespaceDirectory to write files into one folder of the namespace, independently of gitDirectory, which sets the folder to read in the repository. This configuration reads from scripts/ in Git and writes to /shared-scripts in the namespace, so scripts/helpers/parse.py lands at /shared-scripts/helpers/parse.py:

id: sync_shared_scripts
namespace: company.ops
tasks:
- id: sync_files
type: io.kestra.plugin.git.SyncNamespaceFiles
url: https://github.com/my-org/kestra-flows
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
branch: main
namespace: company.team
gitDirectory: scripts
namespaceDirectory: /shared-scripts

With delete: true, only files under namespaceDirectory can be deleted. Files elsewhere in the namespace are kept, even if they aren’t in Git.

Namespace creation

In Enterprise Edition, if the target namespace doesn’t exist, SyncNamespaceFiles creates it before writing files. This needs the Kestra API credentials from Before you begin. Without credentials, the task logs a warning and still writes the files to internal storage, but the namespace doesn’t appear in the UI until you create it. Dry runs never create namespaces.

Open-Source has no API to create namespaces, so the files are synced but the namespace isn’t created.

Sync on a schedule

Add a Schedule trigger to keep Kestra in step with Git. This flow syncs flows and namespace files at the start of every hour:

id: sync_from_git
namespace: company.ops
tasks:
- id: sync_flows
type: io.kestra.plugin.git.SyncFlows
url: https://github.com/my-org/kestra-flows
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
branch: main
targetNamespace: company.team
gitDirectory: flows
- id: sync_files
type: io.kestra.plugin.git.SyncNamespaceFiles
url: https://github.com/my-org/kestra-flows
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
branch: main
namespace: company.team
gitDirectory: _files
triggers:
- id: every_hour
type: io.kestra.plugin.core.trigger.Schedule
cron: "0 * * * *"

Sync on every push

To sync as soon as changes land in Git, trigger the flow with a Webhook trigger and register its URL as a webhook in your Git provider. Store the webhook key as a secret:

id: sync_from_git_on_push
namespace: company.ops
tasks:
- id: sync_flows
type: io.kestra.plugin.git.SyncFlows
url: https://github.com/my-org/kestra-flows
username: git_username
password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
branch: main
targetNamespace: company.team
gitDirectory: flows
triggers:
- id: github_push
type: io.kestra.plugin.core.trigger.Webhook
key: "{{ secret('WEBHOOK_KEY') }}"

In GitHub, open the repository’s Settings > Webhooks > Add webhook, and set Payload URL to the flow’s webhook URL:

https://<your_kestra_host>/api/v1/<tenant>/executions/webhook/company.ops/sync_from_git_on_push/<webhook_key>

In Open-Source, the tenant is main. You can also copy the URL from the flow’s Triggers tab. Your Kestra host must be reachable from GitHub; to test from a local instance, expose it with a tunnel such as ngrok. To reject requests that don’t come from GitHub, set a webhook secret in GitHub and verify the request signature on the trigger.

After you save the webhook, push a commit to the repository. A new execution of sync_from_git_on_push starts and applies the change.

Defaults

PropertySyncFlows defaultSyncNamespaceFiles default
branchmainmain
gitDirectory_flows_files
Target namespacetargetNamespace: requirednamespace: the namespace of the flow that runs the task
includeChildNamespacesfalsefalse
deletefalsefalse
dryRunfalsefalse

If branch or gitDirectory doesn’t exist in the repository, the sync fails. For every property, see the SyncFlows and SyncNamespaceFiles plugin pages.

Was this page helpful?