Copacetic Container Vulnerability Hotpatch Engine
In-place container OS package hotpatching with SecOps approval gates and post-patch verification.
Detect Pulumi pending operations and a stale lock after an interrupted update, clear them with cancel and refresh behind an approval, then verify.
An interrupted pulumi up, from a killed CI job, a dropped connection or a crashed CLI, stops partway and leaves the stack with pending_operations recorded in its deployment and often a held lock. The next pulumi up then refuses with "the stack is currently locked", so every later deploy is blocked until a human intervenes.
This blueprint reads the stack with pulumi stack export, counts the pending operations and any lock file, then reports. When you turn remediation on and approve the pause, it runs pulumi cancel to drop the lock and pulumi refresh to reconcile state with the provider, which clears the stale pending operations. Then it exports the stack again and fails the run if any pending operation or lock survived.
This blueprint was created by zkasuran.
Recovering a stack touches real state, so the flow is built so it cannot change anything by accident:
cancel and refresh need auto_remediate true, dry_run false and an approved pause, all three.pulumi cancel removes the stack lock, it does not undo a resource. The flow states this at the pause so nobody expects a rollback.pulumi refresh reconciles state with what the provider actually has. Plain refresh does not clear a pending create for a resource that never finished, so the flow passes --clear-pending-creates to drop those records. If an interrupted create did reach the provider, import that resource instead of dropping it. The pause is where an operator reads the state and decides.detect (io.kestra.plugin.scripts.shell.Commands on a pulumi/pulumi task runner, read-only) logs in to the self-managed backend with file://, selects the stack, runs pulumi stack export and counts pending_operations and the lock files under .pulumi/locks. It emits the counts and carries the project directory forward as backend.tgz.summary logs the counts and a short list of the pending operations.route stops with a log line when the stack is below the threshold and holds no lock. Otherwise findings_alert posts the state to Slack and remediate_gate checks auto_remediate and dry_run.hold (io.kestra.plugin.core.flow.Pause) waits for approval. On approval remediate runs pulumi cancel --yes then pulumi refresh --yes --clear-pending-creates on the carried backend.reverify exports the stack again and counts pending operations and locks. verify fails the run when either is still above zero, otherwise recovered logs the before and after.errors alerts Slack when the flow fails, including when the verification gate fails.cancel and refresh run only after a human reads the state and approves.file://, S3, Azure Blob, GCS) who has had a deploy blocked by a stale lock or pending operations.The manual fix is to SSH in, read the state, run pulumi cancel and pulumi refresh by hand and hope the operator understood that refresh is not a rollback. That is error prone and unaudited. Kestra reads the stack, posts the state, holds the change behind an approval where the operator reads the refresh implications, runs cancel and refresh, then reads the stack again and fails the execution when pending operations survived. The execution history is the record of what was wedged, who approved the recovery and whether it worked.
On Kestra 2.0.5 OSS with the pulumi/pulumi task runner and a file:// self-managed backend, with a local HTTP sink standing in for Slack. The stack was wedged by interrupting a real pulumi up so its deployment carried pending operations and a lock.
| Run | Inputs | Result |
|---|---|---|
| 1 | clean stack, defaults | 0 pending operations, 0 locks, below threshold. Nothing changed |
| 2 | wedged stack, defaults | pending operations and a lock detected, Slack alerted, nothing changed |
| 3 | wedged, auto_remediate: true |
same plan, nothing changed. The dry run blocks the change even with auto_remediate on |
| 4 | wedged, auto_remediate: true, dry_run: false, pause approved |
cancel dropped the lock, refresh cleared the pending operations, re-export confirmed 0 and 0 |
| 5 | same, pause resumed with approved: false |
nothing changed |
| 6 | same, pause resumed with no answer | nothing changed, an unanswered pause does not read as approval |
| 7 | a re-verify that still found a pending operation | verify failed the run and errors posted to Slack |
| 8 | a backend snapshot that cannot be read | errors alert sent |
file:// backend carried as a snapshot, which needs no cloud credential. For a shared backend, point the login at your s3://, azblob:// or gs:// URL and add its credentials as secrets.pulumi/pulumi container image, pulled by the task runner on first use.SLACK_WEBHOOK_URL: used by findings_alert and the errors block. In Kestra OSS, provide it as the base64 encoded environment variable SECRET_SLACK_WEBHOOK_URL.| Input | Default | Purpose |
|---|---|---|
project_name |
recovery-demo |
The Pulumi project the stack belongs to |
stack_name |
dev |
The stack to inspect and recover |
min_pending_operations |
1 |
Detection threshold for pending operations |
backend_snapshot_b64 |
empty | Optional base64 tar.gz of the project directory and its backend |
auto_remediate |
false |
Must be true to run cancel and refresh |
dry_run |
true |
Must be false to run cancel and refresh |
SLACK_WEBHOOK_URL secret.auto_remediate: true and dry_run: false, read the state in the paused execution, then resume it with approved: true.nightly trigger once the plans look right. Change its inputs when you want it to act.s3://, azblob:// or gs:// URL and loop over several stacks in one run with concurrencyLimit: 1.pulumi up, so a cancelled deploy is recovered right after it fails.creating.