Webhook icon
Schedule icon
WorkingDirectory icon
Commands icon
Docker icon
Script icon
If icon
Create icon
SlackIncomingWebhook icon
Fail icon
Log icon

Block CD Releases on Flux Kustomization Apply Drift and Failing Reconciles

Gate Flux Kustomization apply drift, stale revisions and reconcile failures from one status report, escalating through GitHub issues and Slack alerts.

Categories
Infrastructure

A Flux Kustomization that reports a stale lastAppliedRevision, a climb in reconcile failures, or objects in its inventory that no longer match the manifests will not stop the next CD run by itself, and the release keeps building on the broken base. This blueprint takes one status report from the cluster, materializes it in a shared WorkingDirectory, and grades every kustomization in Python against two thresholds you control. Anything stale, failing or drifted opens a labeled GitHub issue, alerts Slack and fails the execution; a clean fleet logs the in sync position and the run publishes the verdict as outputs.

How it works

  1. post_kustomization_status (io.kestra.plugin.core.trigger.Webhook, keyed with the FLUX_STATUS_WEBHOOK_KEY secret) accepts the report after each reconcile. hourly_drift_safety_net (io.kestra.plugin.core.trigger.Schedule, cron `0 8
      • *) ships disabled: true` until an operator has reviewed the two thresholds and switched it on.
  2. status_workspace (io.kestra.plugin.core.flow.WorkingDirectory) opens one shared directory. materialize_status_report (io.kestra.plugin.scripts.shell.Commands on the Docker runner, image python:3.12-slim) writes the posted body into kustomizations.json with a quoted heredoc, and evaluate_kustomizations (io.kestra.plugin.scripts.python.Script) reads that file plus the two thresholds from env. Both children are scripts inside the workspace, and every non runnable task sits after the block, because default tasks are stateless and a temp dir is purged when a task completes.
  3. The analyst flags a kustomization as stale when lastAppliedRevision differs from lastAttemptedRevision and the attempt is older than stale_revision_hours_allowed (or carries no attempt time at all), as failing when failures passes max_fail_count_allowed or the Ready or Stalled conditions say so, and as drifted when inventory.drift lists objects. The verdict is printed as one ::{"outputs": ...}:: line.
  4. apply_drift_gate (io.kestra.plugin.core.flow.If) is the verdict. A blocked run opens the GitHub issue, posts the Slack summary and then fails through io.kestra.plugin.core.execution.Fail; a clean run logs the count and the failure high water mark. The flow level errors handler (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) reports any other failure with the execution id.

What you get

  • A release gate that reads what Flux reads, revision lag, failure counters and inventory drift, not a hand list.
  • Stale, failing and drifted findings stay separate in the outputs, each with its own fix.
  • A fail safe posture, an unreadable or empty report blocks the release rather than waving it through.
  • Escalation with an artifact, a labeled issue with the JSON lists, a Slack summary and a failing run.

Who it's for

  • GitOps teams running Flux in production who need CD to stop on a broken base instead of layering commits on top of it.
  • Platform engineers who own kustomization hygiene and want failures surfaced with numbers, not screenshots.
  • Release engineers who want one gate output to branch on before promoting across clusters.

Why orchestrate this with Kestra

Flux reconciles and reports continuously but never says whether the next release may proceed, and its status lives one kustomization at a time with no history of yesterday's failure count. Kestra gives the check a keyed webhook for the cluster reporter, a WorkingDirectory where the posted report and the analysis share one file, two thresholds as typed inputs, a conditional gate that escalates into GitHub and Slack and then fails, and an execution history with the exact verdict a promotion pipeline can branch on. The alternative is a script in a cron job whose failure nobody sees until the deploy is already out.

Prerequisites

  • A worker with Docker access, both workspace children run in python:3.12-slim.
  • A reporter step that POSTs flux get kustomizations -o json output shaped as a kustomizations array, each entry carrying metadata.name, spec, status.lastAppliedRevision, status.lastAttemptedRevision, status.lastAttemptedTime, status.failures, status.conditions and status.inventory.drift.
  • A Slack incoming webhook and a GitHub token that can open issues in github_repository.

Secrets

  • FLUX_STATUS_WEBHOOK_KEY: key guarding the status webhook URL, anyone holding it can post a report.
  • SLACK_WEBHOOK_URL: Slack incoming webhook URL used by the drift summary and the errors handler.
  • GITHUB_TOKEN: token used to open the drift issue, classic repo scope or fine-grained Issues write.

Quick start

  1. Add the three secrets and set github_repository to a repository where issues are welcome.

  2. Run the flow once with an in sync report and watch it log the clear verdict.

    curl -X POST -H "Content-Type: application/json" -d '{"observedAt":"2026-10-15T06:00:00Z","kustomizations":[{"metadata":{"name":"infra-prod"},"spec":{"targetRevision":"main"},"status":{"lastAppliedRevision":"main@sha1:9f8e7d6c","lastAttemptedRevision":"main@sha1:9f8e7d6c","lastAttemptedTime":"2026-10-15T05:30:00Z","failures":0,"conditions":[{"type":"Ready","status":"True"}],"inventory":{"entries":[],"drift":[]}}}]}' "https://<kestra>/api/v1/<tenant>/executions/webhook/<namespace>/flux-kustomization-apply-drift-gate/<FLUX_STATUS_WEBHOOK_KEY>"
    
  3. Re-post with lastAppliedRevision behind lastAttemptedRevision and lastAttemptedTime set to 2026-10-14T02:00:00Z to watch the stale branch open the issue, alert Slack and fail.

  4. Raise max_fail_count_allowed above the report's failure count to see the failing branch go quiet, then point the cluster reporter at the webhook and enable hourly_drift_safety_net as the net.

How to extend

  • Add an io.kestra.plugin.core.flow.Loop before the gate that calls io.kestra.plugin.core.http.Request against the Kubernetes API to capture the live object that differs, and attach it to the issue.
  • Store each run's max_fail_count in the Kestra KV store and escalate only on a count that is higher than the previous run, so a steady failure does not page forever.
  • Fan the check across clusters with io.kestra.plugin.core.flow.Loop over a cluster list, each subflow posting its own report.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.