Webhook icon
Dag icon
Request icon
Get icon
Loop icon
Queries icon
Log icon
If icon
Fail icon
Pause icon
OutputValues icon
Set icon
Assert icon
SlackIncomingWebhook icon

Promote Kestra flows from staging to production with approval, snapshot and rollback

Promote Kestra flows between namespaces without Git. Per-flow diff, validation, approval, TTL snapshot, smoke tests, automatic rollback and audit CSV.

Categories
CoreInfrastructure

Many Kestra teams build in a staging namespace and copy flows to prod by hand, or with an export and import that overwrites everything at once. Nobody sees what changes, a flow can reach production with a plugin the server reports as replaced, a hotfix made in prod is silently overwritten, and when a release breaks there is no saved state to go back to. This flow is the release process for that setup: it promotes flows between namespaces through the Kestra REST API, without Git. Every diff, validation, deploy and rollback is a real API call on the instance it runs against.

How it works

  1. A manual run, or the staging_changed webhook from CI, starts the flow. Flow checks refuse empty or wildcard namespaces, a target containing the promoter and a rollback without a snapshot id.
  2. export (a Dag) lists both namespaces and reads the stored snapshot and the promotion state, then fetch_sources reads each flow's YAML with ?source=true.
  3. diff (DuckDB) rewrites every reference to the source namespace (the namespace: line, Subflow calls, Flow triggers) and classifies each flow as new, changed, unchanged or only in the target. It counts changed lines, lists added, removed and modified tasks and triggers, and flags flows edited directly in the target since the last promotion.
  4. validate sends all candidates to /flows/validate in one request. review blocks the run on a constraint error, a deprecated property, a plugin the server reports as replaced, or a Subflow that exists neither in the target nor in the batch. log_report prints the readable diff.
  5. request_approval (a Pause) shows one preselected option per flow with its diff summary. The approver unticks flows to hold them back, or rejects with a reason.
  6. apply lists the target again (optimistic lock), saves a snapshot to KV with a 30-day ttl, deploys one flow at a time, then runs the optional smoke-test executions. Any failure restores every attempted flow from the snapshot.
  7. finally writes an audit CSV, errors alerts with the failed task and an execution link, and afterExecution reports the final state.

Edge cases handled

  • Namespace references are rewritten and counted per flow; the flows' own {{ }} expressions are passed as data and never rendered.
  • A flow that no longer parses (for example saved as a draft with a typo in a task type) is reported and blocks the run.
  • On 2.0.5 the DuckDB plugin reports its unset connectionPooling defaults as deprecated paths for every flow, so only deprecated properties actually written in the source block.
  • A new Subflow called by a ticked flow must be ticked too, otherwise nothing is deployed.
  • A target flow edited while the approval is pending stops the run before any change.
  • A prod hotfix is flagged TARGET EDITED OUTSIDE PROMOTION, so the approver can hold it back.
  • Flows that exist only in the target are reported and never deleted by a promotion. Whitespace-only changes are labelled.

What this blueprint teaches

  • Flow checks (checks): reject unsafe scopes before an execution is created. Their messages are static: checks messages are not rendered.
  • Dag (export): list_source, list_target, load_snapshot and load_state start together; fetch_sources waits for both lists through dependsOn.
  • Pause with a dynamic MULTISELECT (request_approval): expression and defaults render the diff summary as preselected options, so the approver picks a subset.
  • Loop as a transaction (deploy, smoke_tests, restore): concurrencyLimit: 1 with transmitFailed: true stops at the first failure, allowFailure keeps the run going, and restore (transmitFailed: false) undoes exactly what was attempted. Loop outputs keep one row per flow, failed ones included.
  • KV ttl (backup): snapshots expire on their own after 30 days; promotion_state keeps the last deployed hash per flow.
  • finally (write_audit): one CSV row per flow after a success, a dry run, a block, a rejection or a rollback.
  • errors, sla and afterExecution (log_failure, max_runtime, notify_outcome): on 2.0.5 an SLA breach fails the run at once and skips finally; afterExecution still reports it (label promotion_sla: breached).
  • concurrency (limit: 1, FAIL): a second promotion fails at once instead of interleaving deploys.

How this differs from the GitOps blueprints

kestra-gitops-deploy-smoke-test-rollback and git-sync-flows-source-of-truth sync a Git branch with the Git and Kestra plugins, and copy-flows-to-new-tenant imports a whole export. This flow needs no Git: it promotes namespace to namespace, rewrites namespace references, validates each flow against the target, lets the approver pick flows, keeps a snapshot that can still be restored days later with rollback_to_snapshot, never deletes in promote mode, and writes an audit CSV.

What you get

  • A readable per-flow diff and a validation verdict before anything changes (diff_report output).
  • One-by-one deploys with automatic rollback, and a manual rollback mode that is itself reversible.
  • An audit CSV (audit_csv) with status, changed lines, revisions before and after, smoke-test state, outcome, snapshot id and approver reason. Re-runs are idempotent.

Who it's for

  • Platform teams running Kestra with staging and production namespaces, who want reviewed releases without a Git workflow.

Why orchestrate this with Kestra

The release becomes one execution with an approval form, retries and timeouts on every API call, a stored rollback point and an audit file, instead of a script that copies YAML.

Prerequisites

  • Kestra 2.0 with the DuckDB and Slack plugins, and a basic-auth user allowed to read, create, update and delete flows in both namespaces and to start executions in the target.
  • kestra_api_url reachable from the worker. Smoke-test flows should be safe to run with their default inputs (they get the label promotion:smoke-test).

Secrets

  • KESTRA_API_USERNAME and KESTRA_API_PASSWORD: the Kestra API user (required).
  • PROMOTION_WEBHOOK_KEY: key of the staging_changed webhook (required for the trigger).
  • SLACK_WEBHOOK_URL: alerts and outcome messages (optional; every message is also logged).

Quick start

  1. Add the secrets, set source_namespace and target_namespace, keep dry_run: true and run the flow. Read log_report.
  2. Run with dry_run: false, review the options in the approval form and resume.
  3. Note the snapshot id in log_done or in the audit CSV, and restore it later with mode: rollback_to_snapshot if needed.

How to extend

  • Promote namespace files with the same snapshot pattern.
  • Point kestra_api_url at another instance to promote between instances.
  • Post diff_report to a pull request or a change ticket.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.