Commands icon
Docker icon
Log icon
If icon
SlackIncomingWebhook icon
Pause icon
Fail icon
Schedule icon

Recover a Pulumi Stack Wedged by an Interrupted Update

Detect Pulumi pending operations and a stale lock after an interrupted update, clear them with cancel and refresh behind an approval, then verify.

Categories
Infrastructure

An interrupted pulumi up, from a killed CI job, a dropped connection or a crashed CLI, stops partway and leaves the stack with pending_operations recorded in its deployment and often a held lock. The next pulumi up then refuses with "the stack is currently locked", so every later deploy is blocked until a human intervenes.

This blueprint reads the stack with pulumi stack export, counts the pending operations and any lock file, then reports. When you turn remediation on and approve the pause, it runs pulumi cancel to drop the lock and pulumi refresh to reconcile state with the provider, which clears the stale pending operations. Then it exports the stack again and fails the run if any pending operation or lock survived.

This blueprint was created by zkasuran.

Safety

Recovering a stack touches real state, so the flow is built so it cannot change anything by accident:

  • Read-only by default. The shipped inputs and the disabled schedule only export the stack and report. cancel and refresh need auto_remediate true, dry_run false and an approved pause, all three.
  • cancel is not a rollback. pulumi cancel removes the stack lock, it does not undo a resource. The flow states this at the pause so nobody expects a rollback.
  • refresh reconciles, clearing pending creates drops them. pulumi refresh reconciles state with what the provider actually has. Plain refresh does not clear a pending create for a resource that never finished, so the flow passes --clear-pending-creates to drop those records. If an interrupted create did reach the provider, import that resource instead of dropping it. The pause is where an operator reads the state and decides.
  • A snapshot, not the live backend. The flow operates on a snapshot of the project directory carried between steps, so the human approval pause never holds a lock on the live backend.
  • Verify, do not trust. After cancel and refresh the flow exports the stack again and fails the run if a pending operation or lock is still there, so a stuck stack cannot read as a recovered one.

How it works

  1. detect (io.kestra.plugin.scripts.shell.Commands on a pulumi/pulumi task runner, read-only) logs in to the self-managed backend with file://, selects the stack, runs pulumi stack export and counts pending_operations and the lock files under .pulumi/locks. It emits the counts and carries the project directory forward as backend.tgz.
  2. summary logs the counts and a short list of the pending operations.
  3. route stops with a log line when the stack is below the threshold and holds no lock. Otherwise findings_alert posts the state to Slack and remediate_gate checks auto_remediate and dry_run.
  4. hold (io.kestra.plugin.core.flow.Pause) waits for approval. On approval remediate runs pulumi cancel --yes then pulumi refresh --yes --clear-pending-creates on the carried backend.
  5. reverify exports the stack again and counts pending operations and locks. verify fails the run when either is still above zero, otherwise recovered logs the before and after.
  6. errors alerts Slack when the flow fails, including when the verification gate fails.

What you get

  • A single place that finds a stack wedged by an interrupted update and reports it, instead of a developer hitting "the stack is currently locked" on the next deploy.
  • A recovery that is gated and verified, so cancel and refresh run only after a human reads the state and approves.
  • A run that fails when the recovery did not clear the pending operations, so a stuck stack cannot pass as clean.

Who it's for

  • Platform and DevOps teams running Pulumi from CI, where a cancelled or timed out job is the common cause of a wedged stack.
  • Anyone operating self-managed Pulumi backends (file://, S3, Azure Blob, GCS) who has had a deploy blocked by a stale lock or pending operations.

Why orchestrate this with Kestra

The manual fix is to SSH in, read the state, run pulumi cancel and pulumi refresh by hand and hope the operator understood that refresh is not a rollback. That is error prone and unaudited. Kestra reads the stack, posts the state, holds the change behind an approval where the operator reads the refresh implications, runs cancel and refresh, then reads the stack again and fails the execution when pending operations survived. The execution history is the record of what was wedged, who approved the recovery and whether it worked.

Tested end to end

On Kestra 2.0.5 OSS with the pulumi/pulumi task runner and a file:// self-managed backend, with a local HTTP sink standing in for Slack. The stack was wedged by interrupting a real pulumi up so its deployment carried pending operations and a lock.

Run Inputs Result
1 clean stack, defaults 0 pending operations, 0 locks, below threshold. Nothing changed
2 wedged stack, defaults pending operations and a lock detected, Slack alerted, nothing changed
3 wedged, auto_remediate: true same plan, nothing changed. The dry run blocks the change even with auto_remediate on
4 wedged, auto_remediate: true, dry_run: false, pause approved cancel dropped the lock, refresh cleared the pending operations, re-export confirmed 0 and 0
5 same, pause resumed with approved: false nothing changed
6 same, pause resumed with no answer nothing changed, an unanswered pause does not read as approval
7 a re-verify that still found a pending operation verify failed the run and errors posted to Slack
8 a backend snapshot that cannot be read errors alert sent

Prerequisites

  • A Pulumi self-managed backend the task runner can read. This blueprint uses a file:// backend carried as a snapshot, which needs no cloud credential. For a shared backend, point the login at your s3://, azblob:// or gs:// URL and add its credentials as secrets.
  • The pulumi/pulumi container image, pulled by the task runner on first use.

Secrets

  • SLACK_WEBHOOK_URL: used by findings_alert and the errors block. In Kestra OSS, provide it as the base64 encoded environment variable SECRET_SLACK_WEBHOOK_URL.

Inputs

Input Default Purpose
project_name recovery-demo The Pulumi project the stack belongs to
stack_name dev The stack to inspect and recover
min_pending_operations 1 Detection threshold for pending operations
backend_snapshot_b64 empty Optional base64 tar.gz of the project directory and its backend
auto_remediate false Must be true to run cancel and refresh
dry_run true Must be false to run cancel and refresh

Quick start

  1. Add the SLACK_WEBHOOK_URL secret.
  2. Import the flow and run it with the defaults against a snapshot of a wedged stack. Read the pending operations and lock count in the summary.
  3. Run it with auto_remediate: true and dry_run: false, read the state in the paused execution, then resume it with approved: true.
  4. Enable the nightly trigger once the plans look right. Change its inputs when you want it to act.

How to extend

  • Point the backend login at a shared s3://, azblob:// or gs:// URL and loop over several stacks in one run with concurrencyLimit: 1.
  • Trigger the flow from the CI job that runs pulumi up, so a cancelled deploy is recovered right after it fails.
  • Keep the exported state in object storage before refresh, so a failed recovery can be compared against the state as it was.
  • Replace the approval pause with a stricter rule, for example only auto recover a stack whose pending operations are all of type creating.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.