Sync Flows and Namespace Files from Git into Kestra
For the complete documentation index, see llms.txt. For a full content snapshot, see llms-full.txt. Append.mdto anykestra.io/docs/*URL for plain Markdown.
Apply flows and namespace files from a Git repository to Kestra, with Git as the source of truth.
The SyncFlows and SyncNamespaceFiles tasks read a Git branch and create or update the matching flows and namespace files in Kestra. A changed resource in Git overwrites the version in Kestra, including edits made in the UI. To commit resources from Kestra to Git instead, see Push to Git.
Before you begin
- A Git repository with your flows or namespace files, and a personal access token (PAT) with read access to it. Store the PAT as a secret, for example
GITHUB_ACCESS_TOKEN. - Credentials for the Kestra API. Both tasks create and update resources through the API, so on an instance with authentication they need either task-level
auth(andkestraUrlif the API isn’t reachable at the default URL) or default credentials configured withkestra.tasks.sdk.authentication.*, or at the namespace or tenant level in Enterprise Edition. See SDK authentication.
Sync flows
This flow syncs the flows in the flows directory of the main branch into the company.team namespace. With dryRun: true, it computes the changes without applying them:
id: sync_flows_from_gitnamespace: company.ops
tasks: - id: sync_flows type: io.kestra.plugin.git.SyncFlows url: https://github.com/my-org/kestra-flows username: git_username password: "{{ secret('GITHUB_ACCESS_TOKEN') }}" branch: main targetNamespace: company.team gitDirectory: flows dryRun: trueRun the flow and open the task’s Outputs tab. The flows output is a file with one row per flow that the sync adds, updates, or deletes. Check it, then set dryRun: false and run the flow again to apply the changes.
targetNamespaceis required. Every synced flow is written to this namespace, whatevernamespaceits YAML file declares.- Flows that fail validation stop the sync. Set
ignoreInvalidFlows: trueto log a warning and skip them instead. - By default,
deleteisfalse, so flows that exist in Kestra but not in Git are kept. Setdelete: trueto remove them, and run a dry run first to see what would be deleted.
Include child namespaces
Set includeChildNamespaces: true to sync subfolders of gitDirectory into child namespaces of targetNamespace. With targetNamespace: company.team, a flow at flows/tutorial/hello_world_1.yml is synced into company.team.tutorial. This matches the layout that PushFlows creates.
Sync namespace files
SyncNamespaceFiles works the same way for namespace files. This flow syncs the _files directory into the company.team namespace:
id: sync_files_from_gitnamespace: company.ops
tasks: - id: sync_files type: io.kestra.plugin.git.SyncNamespaceFiles url: https://github.com/my-org/kestra-flows username: git_username password: "{{ secret('GITHUB_ACCESS_TOKEN') }}" branch: main namespace: company.team gitDirectory: _files dryRun: true- The dry-run diff is in the task’s
filesoutput. - By default,
deleteisfalse. Setdelete: trueto remove namespace files that aren’t in Git. - With
includeChildNamespaces: true, a folder undergitDirectorynamed after an existing child namespace, such as_files/company.team.tutorial/, is synced into that namespace. This matches the layout thatPushNamespaceFilescreates.
Sync one folder of a namespace
Use namespaceDirectory to write files into one folder of the namespace, independently of gitDirectory, which sets the folder to read in the repository. This configuration reads from scripts/ in Git and writes to /shared-scripts in the namespace, so scripts/helpers/parse.py lands at /shared-scripts/helpers/parse.py:
id: sync_shared_scriptsnamespace: company.ops
tasks: - id: sync_files type: io.kestra.plugin.git.SyncNamespaceFiles url: https://github.com/my-org/kestra-flows username: git_username password: "{{ secret('GITHUB_ACCESS_TOKEN') }}" branch: main namespace: company.team gitDirectory: scripts namespaceDirectory: /shared-scriptsWith delete: true, only files under namespaceDirectory can be deleted. Files elsewhere in the namespace are kept, even if they aren’t in Git.
Namespace creation
In Enterprise Edition, if the target namespace doesn’t exist, SyncNamespaceFiles creates it before writing files. This needs the Kestra API credentials from Before you begin. Without credentials, the task logs a warning and still writes the files to internal storage, but the namespace doesn’t appear in the UI until you create it. Dry runs never create namespaces.
Open-Source has no API to create namespaces, so the files are synced but the namespace isn’t created.
Sync on a schedule
Add a Schedule trigger to keep Kestra in step with Git. This flow syncs flows and namespace files at the start of every hour:
id: sync_from_gitnamespace: company.ops
tasks: - id: sync_flows type: io.kestra.plugin.git.SyncFlows url: https://github.com/my-org/kestra-flows username: git_username password: "{{ secret('GITHUB_ACCESS_TOKEN') }}" branch: main targetNamespace: company.team gitDirectory: flows
- id: sync_files type: io.kestra.plugin.git.SyncNamespaceFiles url: https://github.com/my-org/kestra-flows username: git_username password: "{{ secret('GITHUB_ACCESS_TOKEN') }}" branch: main namespace: company.team gitDirectory: _files
triggers: - id: every_hour type: io.kestra.plugin.core.trigger.Schedule cron: "0 * * * *"Sync on every push
To sync as soon as changes land in Git, trigger the flow with a Webhook trigger and register its URL as a webhook in your Git provider. Store the webhook key as a secret:
id: sync_from_git_on_pushnamespace: company.ops
tasks: - id: sync_flows type: io.kestra.plugin.git.SyncFlows url: https://github.com/my-org/kestra-flows username: git_username password: "{{ secret('GITHUB_ACCESS_TOKEN') }}" branch: main targetNamespace: company.team gitDirectory: flows
triggers: - id: github_push type: io.kestra.plugin.core.trigger.Webhook key: "{{ secret('WEBHOOK_KEY') }}"In GitHub, open the repository’s Settings > Webhooks > Add webhook, and set Payload URL to the flow’s webhook URL:
https://<your_kestra_host>/api/v1/<tenant>/executions/webhook/company.ops/sync_from_git_on_push/<webhook_key>In Open-Source, the tenant is main. You can also copy the URL from the flow’s Triggers tab. Your Kestra host must be reachable from GitHub; to test from a local instance, expose it with a tunnel such as ngrok. To reject requests that don’t come from GitHub, set a webhook secret in GitHub and verify the request signature on the trigger.
After you save the webhook, push a commit to the repository. A new execution of sync_from_git_on_push starts and applies the change.
Defaults
| Property | SyncFlows default | SyncNamespaceFiles default |
|---|---|---|
branch | main | main |
gitDirectory | _flows | _files |
| Target namespace | targetNamespace: required | namespace: the namespace of the flow that runs the task |
includeChildNamespaces | false | false |
delete | false | false |
dryRun | false | false |
If branch or gitDirectory doesn’t exist in the repository, the sync fails. For every property, see the SyncFlows and SyncNamespaceFiles plugin pages.
Was this page helpful?