DocsVersion Control & CI/CDVersion Control with Git

Version Control with Git​Version ​Control with ​Git

Setup Version Control with Git to store your flows and namespace files.

Kestra supports version control with Git. You can use one or more Git repositories to store your Flows and Namespace Files, and track changes to them over time via Git commit history.

There are multiple ways to use Git with Kestra:

  • The git.SyncFlows pattern allows you to implement GitOps and use Git as a single source of truth for your flows.
  • The git.SyncNamespaceFiles pattern allows you to implement GitOps and use Git as a single source of truth for your namespace files.
  • The git.PushFlows pattern allows you to edit your flows from the UI and regularly commit and push changes to Git; this pattern is useful if you want to use the built-in Editor in the UI and still have your code in Git.
  • The git.PushNamespaceFiles pattern allows you to edit your namespace files from the UI and regularly commit and push changes to Git; this pattern is useful if you want to use the built-in Editor in the UI and still have your files in Git.
  • The CI/CD pattern is useful if you want to manage the CI/CD process yourself e.g. via GitHub Actions or Terraform, and keep Git as a single source of truth for your code.

The image below shows how to choose the right pattern based on your needs:

git

Let's dive into each of these patterns, and when to use them.

Git SyncFlows and SyncNamespaceFiles

The Git SyncFlows pattern implements GitOps and uses Git as a single source of truth. It allows you to store your flows in Git and use a system flow that automatically syncs changes from Git to Kestra. You can also sync namespace files using the Git SyncNamespaceFiles pattern in the same way.

Here's how that works:

  • You store your flows and Namespace Files in Git
  • You create a system flow that runs on a schedule and syncs changes from Git to Kestra
  • When you want to make a change to a flow or a namespace file, you modify the file in Git
  • The system flow syncs changes from Git to Kestra so that even if you make changes to any flows or Namespace Files from the UI, the changes are overwritten by the changes from Git.

This pattern is useful if you want to use Git as a single source of truth and avoid making changes to flows and Namespace Files from the UI. Using this pattern, you don't need to manage any CI/CD pipelines.

If your team follows the GitOps methodology, or you're coming from a Kubernetes background, this pattern is for you.

Here is an example system flow that you can use to declaratively sync changes from Git to Kestra:

yaml
id: sync_from_git
namespace: system

tasks:
  - id: git
    type: io.kestra.plugin.git.SyncFlows
    url: https://github.com/kestra/scripts
    branch: main
    username: git_username
    password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
    targetNamespace: git
    includeChildNamespaces: true # optional; by default, it's set to false to allow explicit definition
    gitDirectory: your_git_dir

triggers:
  - id: schedule
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "*/1 * * * *" # every minute

You can choose to commit this flow to Git or add it from the built-in Editor in Kestra UI — this flow won't be overwritten by the Git reconciliation process.

You can also sync namespace files with the example below:

yaml
id: sync_from_git
namespace: system


tasks:
  - id: git
    type: io.kestra.plugin.git.SyncNamespaceFiles
    namespace: prod
    gitDirectory: _files # optional; set to _files by default
    url: https://github.com/kestra-io/flows
    branch: main
    username: git_username
    password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"

This flow can also be triggered anytime you push changes to Git via a GitHub webhook:

yaml
id: sync_from_git
namespace: system

tasks:
  - id: git
    type: io.kestra.plugin.git.SyncFlows
    url: https://github.com/kestra/scripts
    branch: main
    targetNamespace: git
    username: git_username
    password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"

triggers:
  - id: github_webhook
    type: io.kestra.plugin.core.trigger.Webhook
    key: "{{ secret('WEBHOOK_KEY') }}"

Note that the webhook key is used to authenticate webhook requests and prevent unauthorized access to your Kestra instance. For the above flow, you would paste the following URL in your GitHub repository settings in the Webhooks section:

bash
https://us.kestra.cloud/api/v1/your_tenant/executions/webhook/prod/sync_from_git/your_secret_key

github_webhook

Following the pattern:

bash
https://<host>/api/v1/<tenant>/executions/webhook/<namespace>/<flow>/<webhook_key>

CI/CD

The CI/CD pattern allows you to use Git as a single source of truth and push code changes to Kestra anytime you merge a pull request. However, in contrast to the Git Sync pattern, you need to manage the CI/CD process yourself e.g. via GitHub Actions or Terraform. Check the CI/CD documentation for more details on how to set up CI/CD for Kestra flows and Namespace Files.

Git PushFlows and PushNamespaceFiles

The Git PushFlows pattern allows you to edit your flows from the UI, and regularly push changes to Git. It's particularly helpful if you want to use the built-in Editor in the UI and have your code change history managed via Git. You can also push namespace files using the Git PushNamespaceFiles pattern in the same way.

Here is example flow that you can use to push flow changes from Kestra to Git:

yaml
id: push_to_git
namespace: system

tasks:
  - id: commit_and_push
    type: io.kestra.plugin.git.PushFlows
    url: https://github.com/kestra-io/scripts
    sourceNamespace: dev
    targetNamespace: pod
    flows: "*"
    branch: kestra
    username: github_username
    password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
    commitMessage: add namespace files changes

triggers:
  - id: schedule
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "* */1 * * *" # every hour

Here is an example you can use to push namespace files from Kestra to Git:

yaml
id: push_to_git
namespace: system

tasks:
  - id: commit_and_push
    type: io.kestra.plugin.git.PushNamespaceFiles
    namespace: dev
    files: "*"
    gitDirectory: _files
    url: https://github.com/kestra-io/scripts # required string
    username: git_username
    password: "{{ secret('GITHUB_ACCESS_TOKEN') }}"
    branch: dev
    commitMessage: "add namespace files"


triggers:
  - id: schedule_push_to_git
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "*/15 * * * *"

You can use that pattern to push changes to a feature branch and create a pull request for review. Once the pull request is approved, you can merge it to the main branch.

Git Clone

The Git Clone pattern allows you to clone a Git repository at runtime. This pattern can be used to orchestrate code maintained in a different code repository (potentially managed by a different team) in the following scenarios:

Was this page helpful?