Enable GitOps for Kestra with the SyncFlows Task

Sync flows from a Git Repository to Kestra with the SyncFlows Task.

The SyncFlows task is a powerful integration that allows you to sync your code with Git from the UI while still managing this process entirely in code! Kestra unifies the development experience between the UI and code so you can combine the best of both worlds without sacrificing the benefits of version control.

The task syncs one or more flows from a Git repository on a schedule or anytime you push a change to a given Git branch.

Before you begin

Before you start using the SyncFlows task, ensure the following prerequisites are in place:

1. A Git repository where you want to sync your flows. If you haven’t pushed any flows yet, see the guide using the PushFlows task.
2. A Personal Access Token (PAT) for Git authentication.
3. A running Kestra instance in a version 0.17.0 or later with the PAT stored as a secret within the Kestra instance.

Using the dryRun property

Here is a system flow that will sync the git namespace with flows from the repository in the flows directory.

id: sync_flows_from_git
namespace: system

tasks:
  - id: sync_flows
    type: io.kestra.plugin.git.SyncFlows
    username: git_username
    password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
    url: https://github.com/git_username/flows
    branch: main
    targetNamespace: git
    gitDirectory: flows
    dryRun: true

Given that the dryRun property is set to true, the task will only output changes from the Git repository without syncing any flows to Kestra yet:

The files listed are the same ones we added in the PushFlows guide.

Sync all flows to a single namespace from Git

Set the dryRun property to false and sync the repository with Kestra:

id: sync_flows_from_git
namespace: system

tasks:
  - id: sync_flows
    type: io.kestra.plugin.git.SyncFlows
    ...
    dryRun: false

You should see the same flows from the earlier log now in Kestra:

A full list is also available in the Outputs tab:

Sync all flows including child namespaces

You can also sync all flows in child namespaces. In the repository, there is a sub-folder called tutorial with more flows. Sync those as well by adding the includeChildNamespaces property and setting it to true.

id: sync_flows_from_git
namespace: system

tasks:
  - id: sync_flows
    type: io.kestra.plugin.git.SyncFlows
    username: git_username
    password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
    url: https://github.com/kestra-io/flows
    branch: main
    targetNamespace: git
    gitDirectory: flows
    includeChildNamespaces: true

After executing, all flows — including those from the tutorial child namespace — are synced into Kestra:

The Outputs tab shows the same result:

Set up a schedule

A common use case for this task is to set up a routine schedule to keep Kestra in sync with the Git repository. Add a Schedule trigger. This example has a cron expression to execute once every hour:

id: sync_flows_from_git
namespace: system

tasks:
  - id: sync_flows
    type: io.kestra.plugin.git.SyncFlows
    username: git_username
    password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
    url: https://github.com/git_username/flows
    branch: main
    targetNamespace: git
    gitDirectory: flows

triggers:
  - id: every_full_hour
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "* 0 * * *"

Automatically sync when a change is pushed to Git

You can also automate the syncing process by adding a Webhook trigger and creating a Webhook on your GitHub repository to trigger the flow every time something is pushed to the repository. This is useful for keeping Kestra always in sync with the repository.

id: sync_flows_from_git
namespace: system

tasks:
  - id: sync_flows
    type: io.kestra.plugin.git.SyncFlows
    username: git_username
    password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
    url: https://github.com/kestra-io/flows
    targetNamespace: git
    gitDirectory: flows

triggers:
  - id: gh_webhook
    type: io.kestra.plugin.core.trigger.Webhook
    key: abcdefg

To setup this webhook, go to the Settings for your GitHub repository and head to Webhooks and create a new Webhook:

For the Payload URL, your URL will follow the following format:

https://{your_hostname}/api/v1/main/executions/webhook/system/sync_flows_from_git/abcdefg

This will require your host name to be publicly accessible. If you want to test this without having to deploy Kestra first, you can use a tool like ngrok to tunnel Kestra so GitHub can see it. As we’re putting the secret in the URL, we can leave the Secret field blank.

Save and test by committing something to the Git repository.

The most recent execution was triggered by the Webhook, keeping Kestra in sync with the Git repository automatically.

If you also want to sync your files, see the guide on syncing namespace files.

Extra notes

• The branch property allows you to specify the branch to which files should be synced from.
• The gitDirectory property allows you to specify the directory to which flows should be synced from. If not set, flows will be synced from the Git directory named _flows and will optionally also include subdirectories named after the child namespaces. If you prefer, you can specify an arbitrary path, e.g. kestra/flows, allowing you to sync flows to that specific Git directory.
• If you try to add the Personal Access Token (PAT) directly in your source code in the password property, you will get an error message. This is a safety mechanism to prevent you and your users from accidentally exposing your PAT in the source code. You should store the PAT as a Kestra Secret, environment variable, namespace variable or as a SECRET-type input in your flow.