Schedule icon
Request icon
OutputValues icon
Log icon
Switch icon
SlackIncomingWebhook icon
Pause icon
If icon
Loop icon
CancelProcessInstance icon
Sleep icon
Fail icon

Cancel Camunda Process Instances Stuck on an Unresolvable Incident

Detect Camunda 8 process instances parked on an exhausted-retry incident, cancel them behind an approval gate, then re-read the engine.

Categories
Infrastructure

Safety

Cancelling a process instance is destructive in a way that deleting a stale row is not. The instance loses its variables, it loses its token position, it never reaches an end event, so whatever real-world process it represented is abandoned. Nothing brings it back. An incident with retries left should be retried, not cancelled, which is why this flow reads the job's live retry count and refuses any instance that still has one.

Four controls stand between a finding and a cancellation, each one checkable on its own:

  • a grace window, so an incident someone is working on right now is not a candidate;
  • a process definition allowlist, so only definitions somebody approved can ever be cancelled;
  • a per-run ceiling, so a spike cancels nothing at all and alerts instead;
  • retries at exactly 0, read from the job rather than inferred from the incident.

Then auto_remediate and dry_run and the Pause must all agree. A re-read of the engine decides whether the run passed.

How it works

A Camunda 8 job that burns every retry raises a JOB_NO_RETRIES incident. The instance stops there. Nothing retries it, because there are no retries left. Nothing alerts, because the cluster is healthy. The instance keeps its job, its variables and its row in the runtime tables. The definition it references cannot be cleanly retired while instances still point at it. Teams find hundreds of these months later.

  1. read_stuck_incidents (io.kestra.plugin.core.http.Request) posts to /v2/incidents/search for state: ACTIVE, errorType: JOB_NO_RETRIES, a processDefinitionId $in the allowlist and a creationTime $lt the grace cutoff. Both the grace window and the allowlist are server-side filters, so neither depends on this flow filtering correctly afterwards.
  2. read_exhausted_jobs posts to /v2/jobs/search for state: FAILED with retries: 0. This read is the point of the flow. An incident's errorType records why it was raised, not what the job looks like now, so a job whose retries were pushed back up keeps an open JOB_NO_RETRIES incident while it is perfectly retryable.
  3. read_active_before records the cluster-wide ACTIVE instance count before anything changes.
  4. classify_findings (io.kestra.plugin.core.output.OutputValues) intersects the two reads on processInstanceKey in one jq program and picks a single decision: NO_FINDINGS, OVER_CEILING, REPORT_ONLY or CANCEL. Every number stays inside that one program, so no count is retyped between tasks.
  5. log_findings prints the detector's own numbers, so the execution list shows what the run saw.
  6. route_decision (io.kestra.plugin.core.flow.Switch) takes one branch per decision. Its defaults branch fails the run, because a jq program that errors yields nothing, which would otherwise read as a clean cluster.
  7. On CANCEL, confirm_cancellation (io.kestra.plugin.core.flow.Pause) holds the run until a human answers approved. The condition reads approved ?? false, so resuming with no answer cancels nothing.
  8. cancel_stuck_instances (io.kestra.plugin.core.flow.Loop, concurrencyLimit: 1) runs io.kestra.plugin.camunda.CancelProcessInstance once per approved key.
  9. settle_secondary_storage waits, then reverify_cancelled_keys asks the engine which approved keys are still ACTIVE. guard_reverify fails the run on anything but zero. A cancel command Camunda refused still leaves its task green, so this count is the only trustworthy signal.
  10. The errors block alerts Slack whenever the run fails, which includes a refused cancellation caught by the re-verify.

What you get

  • A detector for a failure mode no Camunda dashboard raises, because the cluster really is healthy: the instance is simply parked forever.
  • Live retry state read from the job, not inferred from the incident, so an instance that should be retried is refused instead of destroyed.
  • Four independent controls (grace window, definition allowlist, per-run ceiling, retries at 0) plus auto_remediate, dry_run and a Pause in front of one destructive command.
  • A re-read of the engine that fails the run when a cancellation did not land, which is what catches Camunda refusing to cancel an instance a call activity created.
  • A per-run audit line, so the execution history becomes the record of how many instances were parked on an incident over time.

Who it's for

  • Platform teams running Camunda 8 who need stuck instances found before a definition cleanup or a cluster migration trips over them.
  • Operations teams who want cancellation to be an approved, auditable act rather than a click in Operate.
  • Anyone who has tried to retire a BPMN definition and discovered that instances nobody owns still reference it.

Why orchestrate this with Kestra

Camunda Operate can list incidents and cancel an instance, one click at a time, with no record of who decided what. The decision here is the hard part, not the command. Kestra supplies the cadence, joins the incident read to the live job read so retryable instances are excluded, holds the allowlist and the ceiling as inputs an operator can review, parks the destructive step on an approval gate, then fails the run when the engine says a cancellation did not land. The execution history is the audit trail.

Tested end to end

Built and run against a real single-node Camunda 8.9.19 Orchestration Cluster with RDBMS secondary storage. Four BPMN definitions were deployed, then instances were seeded through Camunda's own REST API into these states: a job failed with retries: 0 to raise a real JOB_NO_RETRIES incident, the same state with the job's retries pushed back up to 2, an identical incident on a definition outside the allowlist, a healthy ACTIVE instance with no incident, then a child instance created by a call activity.

Every path was run for real. A clean cluster returned zero findings. With three active incidents on the cluster the detector reported exactly one cancellable instance, refusing the retries-remaining instance and the non-allowlisted one. Raising the grace window to 60 minutes returned zero findings from the same cluster state. dry_run: true named the candidate without touching it. An approved run cancelled it, the re-verify confirmed the key was gone. The untouched controls were still ACTIVE in Camunda's own API. A declined Pause cancelled nothing. A finding count over the ceiling cancelled nothing and alerted. The call-activity child reproduced a real refused cancellation: the plugin task went green, the re-verify found the instance still ACTIVE, the run failed, the Slack errors block fired.

Prerequisites

Secrets

  • SLACK_WEBHOOK_URL: Slack incoming webhook for the failure alert and the ceiling alert.
  • Add CAMUNDA_USERNAME and CAMUNDA_PASSWORD to the CancelProcessInstance task if your cluster has API protection enabled. The v2 search reads then need an Authorization header on the Request tasks.

Inputs

  • camunda_rest_address: base URL of the Camunda 8 Orchestration Cluster REST API.
  • allowed_process_definition_ids: JSON array of BPMN process IDs whose instances may be cancelled.
  • grace_minutes: how long an incident must have been open to count. Default 120.
  • max_cancellations: per-run ceiling. Above it the run cancels nothing and alerts. Default 5.
  • auto_remediate: "false" by default. Must be "true" for any cancellation.
  • dry_run: "true" by default. Must be "false" for any cancellation.

The cluster needs secondary storage (Elasticsearch, OpenSearch or RDBMS) because the v2 search endpoints read from it. A broker started with CAMUNDA_DATA_SECONDARYSTORAGE_TYPE: none serves the commands but not the searches.

Quick start

  1. Add the SLACK_WEBHOOK_URL secret.
  2. Run with the defaults. auto_remediate: "false" and dry_run: "true" make it read-only, so the first run only tells you how many instances are parked.
  3. Read log_findings. It prints the candidate keys, their definitions, their incident keys and how long each incident has been open.
  4. Put only the definitions you are willing to abandon into allowed_process_definition_ids.
  5. To cancel, run with auto_remediate: "true" and dry_run: "false", then answer approved: true on the Pause. Answer anything else to walk away.
  6. Enable hourly_stuck_instance_sweep once a manual run looks right. The trigger forces report-only mode on its own inputs.

How to extend

  • Resolve instead of cancel. Swap the Loop body for a POST /v2/incidents/{incidentKey}/resolution request to retry the job, then keep the same re-verify shape for instances that genuinely deserve a retry.
  • Cancel the root instead of the child. Group findings by parentProcessInstanceKey and cancel the root once, which is what Camunda asks for when a call activity created the instance.
  • Capture the variables first. Add a /v2/process-instances/{key}/variables/search read before the Loop and archive the result, so a cancelled instance leaves a record of what it was carrying.
  • Widen the detector. Drop the errorType filter to catch UNHANDLED_ERROR_EVENT and CALLED_ELEMENT_ERROR incidents, which park an instance just as hard.
  • Narrow the window. Add a MULTISELECT of error messages so only known-unrecoverable failures qualify.

Links

This blueprint was created by zkasuran.

See How

New to Kestra?

Use blueprints to kickstart your first workflows.