DocsDeveloper GuideCreate a CI/CD pipeline

Create a CI/CD pipeline​Create a ​C​I/​C​D pipeline

Kestra provides several ways to create a CI/CD pipeline for your flows. This section explains how to automate the validation and deployment of your workflows using CI/CD.

Introduction

Manual deployments are often not desirable. CI/CD pipelines are a great way to automate the validation and deployment of your workflows to production environments. This section explains several ways of creating a CI/CD pipeline for your Kestra resources.

Why a CI/CD pipeline?

A CI/CD process helps ensure fast and reliable deployments. Your changes get deployed automatically, as soon as they get peer-reviewed and merged to a VCS (Version Control System) like Git.

CI/CD for Kestra workflows

There are several ways to create a CI/CD pipeline for your flows in Kestra. Pick one of the options listed below that best suits your needs.

Kestra CLI

Kestra CLI provides several commands for validating and deploying your flows:

bash
# validate a single flow
./kestra flow validate flow_directory/myflow.yml --server http://localhost:8080 --api-token <your-api-token>

# deploy a single flow to a namespace without deleting existing flows
./kestra flow namespace update namespace_name flow_directory/myflow.yml --no-delete --server http://localhost:8080 --api-token <your-api-token>

If you run Kestra in a Docker container, you can access the CLI as follows:

bash
docker exec -it kestra-container-name /bin/bash
./kestra flow --help

To validate and deploy multiple flows within a directory named flows, you can provide a path to that directory:

bash
./kestra flow validate flows/ --server http://localhost:8080 --api-token <your-api-token>
./kestra flow namespace update namespace_name flows/ --no-delete --server http://localhost:8080 --api-token <your-api-token>

The --no-delete flag is used to preserve existing flows already stored within your Kestra instance. Don't include that flag if you want that a specific Git repository or a local directory is used as the single source of truth for your production workflows. This way, all previously stored flows get deleted, and only those flows maintained in Git (or that directory) are kept and updated.

CLI options

The CLI provides several options to customize your validation and deployment process. CLI options you should be aware of include:

  • --local: performs the validation locally using the client. By default, Kestra validates flows server-side (a remote API call to your Kestra webserver/standalone server).
  • --server: allows you to specify the remote Kestra webserver/standalone server URL. The default URL is http://localhost:8080.

For all available CLI options on both flow validate and flow namespace update commands, use the -h or --help flag:

bash
./kestra flow validate -h
./kestra flow namespace update -h

Templates (deprecated)

Templates can be validated and deployed in the same way as flows:

bash
./kestra template validate path-to-template-directory
./kestra template namespace update template-namespace-to-update path-to-template-directory

Deploy flows... from a flow!

The CLI commands explained above can be used in a Shell script within a flow. This way, your CI/CD pipeline can be managed directly from your Kestra instance directly.

yaml
id: ci-cd
namespace: prod

tasks:
  - id: github-ci-cd
    type: io.kestra.core.tasks.flows.WorkingDirectory
    tasks:
      - id: clone-repository
        type: io.kestra.plugin.git.Clone
        url: https://github.com/anna-geller/kestra-ci-cd
        branch: main

      - id: validate-flows
        type: io.kestra.plugin.scripts.shell.Commands
        description: "Validate flows from Git before deploying them."
        runner: PROCESS
        warningOnStdErr: false
        commands:
          - /app/kestra flow validate flows/ --server http://localhost:8080 --api-token "{{ secret('KESTRA_API_TOKEN') }}"

      - id: deploy-flows
        type: io.kestra.plugin.scripts.shell.Commands
        description: "Deply flows to a Kestra namespace."
        runner: PROCESS
        warningOnStdErr: false
        commands:
          - /app/kestra flow namespace update prod flows/prod/ --server http://localhost:8080 --api-token "{{ secret('KESTRA_API_TOKEN') }}"
          - /app/kestra flow namespace update prod.marketing flows/prod.marketing/ --server http://localhost:8080 --api-token "{{ secret('KESTRA_API_TOKEN') }}"

triggers:
  - id: github
    type: io.kestra.core.models.triggers.types.Webhook
    key: "yourSecretKey1234"

While you can trigger that deployment pipeline manually using Kestra UI or API (i.e. just like any other Kestra flow), it's better to combine that with a push event emitted by your Git repository. The above flow leverages the Webhook trigger so that your CI/CD flow runs as soon as you push changes to your Git branch.

To configure the webhook for your GitHub repository, go to Settings, and then select Webhooks. The URL in your browser should look as follows:

bash
https://github.com/your_username/your_repository_name/settings/hooks

Select "Add webhook":

github_webhook_1

Paste Kestra's webhook URL into the Payload URL field, as shown below. The webhook to trigger a Kestra flow should be in the following format:

bash
https://kestra_host_url/api/v1/executions/webhook/namespace/flow_id/webhook_key

github_webhook_2

Note that we configured this webhook to be sent upon a push event to the default branch, but you can choose the option "Let me select individual events" for further customization e.g. to trigger the flow any time there is a new pull request.

Deploy flows from a GitHub Action

The official Kestra's GitHub Actions leverage the same CLI commands to:

  1. Validate flows and templates using the Validate Action
  2. Deploy flows and templates using the Deploy Action.

Here is a full example validating and deploying flows within a GitHub Actions workflow:

yaml
name: Kestra CI/CD
on:
  push:
    branches:
      - main
jobs:
  prod:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: validate-all flows
        uses: kestra-io/validate-action@develop
        with:
          directory: ./flows/prod
          resource: flow
          server: ${{secrets.KESTRA_HOSTNAME}}
          user: ${{secrets.KESTRA_USER}}
          password: ${{secrets.KESTRA_PASSWORD}}
      - name: deploy-prod
        uses: kestra-io/deploy-action@develop
        with:
          namespace: prod
          directory: ./flows/prod
          resource: flow
          server: ${{secrets.KESTRA_HOSTNAME}}
          user: ${{secrets.KESTRA_USER}}
          password: ${{secrets.KESTRA_PASSWORD}}
          delete: false
      - name: deploy-prod-marketing
        uses: kestra-io/deploy-action@develop
        with:
          namespace: prod.marketing
          directory: ./flows/prod.marketing
          resource: flow
          server: ${{secrets.KESTRA_HOSTNAME}}
          user: ${{secrets.KESTRA_USER}}
          password: ${{secrets.KESTRA_PASSWORD}}
          delete: false

Note that this example uses GitHub repository secrets to store the host name, user name and password to your Kestra instance. Make sure to add those secrets to your repository before using this workflow.