id: camunda-incident-stuck-process-instance-cancel-guard
namespace: company.team
description: |
Find Camunda 8 process instances parked on an unresolvable incident, then cancel
them behind an approval gate and prove from the engine that they are gone.
An instance whose job has burned every retry raises a JOB_NO_RETRIES incident and
then stops moving. The engine stays healthy, so nothing alerts, while the instance
keeps its job, its variables and its row in the runtime tables. The process
definition cannot be cleanly retired while instances still reference it.
Cancelling destroys in-flight business state. Four controls stand in front of the
write: a grace window, a process-definition allowlist, a per-run ceiling, plus a
live re-read of the job's own retries so an instance that still has retries left is
refused rather than cancelled. Read-only by default.
triggers:
- id: hourly_stuck_instance_sweep
type: io.kestra.plugin.core.trigger.Schedule
description: Sweep for stuck instances every hour. Shipped disabled. Enable it
once camunda_rest_address plus allowed_process_definition_ids point at a
real cluster and one manual run has been checked. Its own inputs force
report-only mode whatever this flow's defaults say.
cron: "17 * * * *"
disabled: true
inputs:
auto_remediate: "false"
dry_run: "true"
inputs:
- id: camunda_rest_address
type: STRING
displayName: "Camunda REST API Base URL"
defaults: "http://localhost:8080"
description: Base URL of the Camunda 8 Orchestration Cluster REST API. Used for
the v2 search reads and passed as restAddress to the CancelProcessInstance
task.
- id: allowed_process_definition_ids
type: STRING
displayName: "Cancellable Process Definition IDs (JSON array)"
defaults: "[\"order-fulfillment\"]"
description: >-
JSON array of BPMN process IDs (Camunda's processDefinitionId) whose
instances may be cancelled. The list is pushed into the incident search as
an $in filter, so a definition that is not named here is never even a
candidate. Keep it short. Only put a definition here once someone has
agreed that abandoning an in-flight instance of it is acceptable.
- id: grace_minutes
type: INT
displayName: "Incident Grace Window (minutes)"
defaults: 120
description: An incident must have been open at least this long before its
instance counts as stuck. Protects an incident someone is working on right
now. It also spares one a worker is about to clear.
- id: max_cancellations
type: INT
displayName: "Cancellations Per Run Ceiling"
defaults: 5
description: If a run finds more cancellable instances than this, it cancels
nothing and alerts instead. A sudden large finding count usually means a
bad deployment or a dead worker pool, not a backlog to delete.
- id: auto_remediate
type: SELECT
displayName: "Auto Remediate"
defaults: "false"
values:
- "false"
- "true"
description: Set to "true" to allow cancellation at all. With "false" the run
reports what it found.
- id: dry_run
type: SELECT
displayName: "Dry Run"
defaults: "true"
values:
- "true"
- "false"
description: Must be explicitly "false", together with auto_remediate "true",
before any instance is cancelled. Either gate alone keeps the run
read-only.
tasks:
- id: read_stuck_incidents
type: io.kestra.plugin.core.http.Request
description: >-
Active JOB_NO_RETRIES incidents older than the grace window, restricted to
the allowed definitions. Camunda's v2 search API does the grace window
with a creationTime $lt filter and the allowlist with a
processDefinitionId $in filter, so neither control depends on this flow
filtering correctly afterwards.
uri: "{{ inputs.camunda_rest_address }}/v2/incidents/search"
method: POST
contentType: application/json
body: |
{
"filter": {
"state": "ACTIVE",
"errorType": "JOB_NO_RETRIES",
"processDefinitionId": {"$in": {{ inputs.allowed_process_definition_ids }} },
"creationTime": {"$lt": "{{ execution.startDate | dateAdd(0 - inputs.grace_minutes, 'MINUTES') | date("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", timeZone="UTC") }}"}
},
"page": {"limit": 1000}
}
- id: read_exhausted_jobs
type: io.kestra.plugin.core.http.Request
description: >-
Jobs that are FAILED with retries at exactly 0. An incident's errorType
records why the incident was raised, not what the job looks like now: a
job whose retries were pushed back up keeps an open JOB_NO_RETRIES
incident. So retries are read from the job itself, live. Only an instance
that appears in both reads is treated as stuck.
uri: "{{ inputs.camunda_rest_address }}/v2/jobs/search"
method: POST
contentType: application/json
body: |
{
"filter": {"state": "FAILED", "retries": 0},
"page": {"limit": 1000}
}
- id: read_active_before
type: io.kestra.plugin.core.http.Request
description: Count of every ACTIVE process instance on the cluster before
anything changes. The headline number the audit line compares against.
uri: "{{ inputs.camunda_rest_address }}/v2/process-instances/search"
method: POST
contentType: application/json
body: |
{
"filter": {"state": "ACTIVE"},
"page": {"limit": 1}
}
- id: classify_findings
type: io.kestra.plugin.core.output.OutputValues
description: >-
Intersect the two reads on processInstanceKey, then pick one decision for
the whole run. NO_FINDINGS, OVER_CEILING, REPORT_ONLY or CANCEL.
Everything numeric happens inside the single jq program, so no count is
retyped between tasks.
values:
result: >-
{{ {'incidents': outputs.read_stuck_incidents.body, 'jobs':
outputs.read_exhausted_jobs.body, 'ceiling': inputs.max_cancellations,
'auto_remediate': inputs.auto_remediate, 'dry_run': inputs.dry_run} |
toJson
| jq('. as $r | ($r.incidents | fromjson) as $i | ($r.jobs | fromjson) as $j | [$j.items[] | .processInstanceKey | tostring] as $ex | [$i.items[] | . as $inc | select(([$ex[] | select(. == ($inc.processInstanceKey | tostring))] | length) > 0)] as $hits | ($hits | length) as $n | {count: $n, keys: [$hits[] | .processInstanceKey | tostring], detail: ([$hits[] | ((.processInstanceKey | tostring) + " " + (.processDefinitionId | tostring) + " incident=" + (.incidentKey | tostring) + " opened=" + (.creationTime | tostring))] | join("; ")), ceiling: $r.ceiling, decision: (if $n == 0 then "NO_FINDINGS" elif $n > $r.ceiling then "OVER_CEILING" elif ($r.auto_remediate == "true" and $r.dry_run == "false") then "CANCEL" else "REPORT_ONLY" end)} | tojson')
| first }}
- id: log_findings
type: io.kestra.plugin.core.log.Log
description: One line carrying the detector's own numbers, so the execution list
shows what each run saw without opening a task output.
message: >-
Stuck instance scan on {{ inputs.camunda_rest_address }}:
active_instances={{ outputs.read_active_before.body |
jq('.page.totalItems') | first }} incidents_past_grace={{
outputs.read_stuck_incidents.body | jq('.page.totalItems') | first }}
jobs_retries_zero={{ outputs.read_exhausted_jobs.body |
jq('.page.totalItems') | first }} cancellable={{
outputs.classify_findings.values.result | jq('.count') | first }}
ceiling={{ inputs.max_cancellations }} grace_minutes={{
inputs.grace_minutes }} decision={{
outputs.classify_findings.values.result | jq('.decision') | first }}
detail=[{{ outputs.classify_findings.values.result | jq('.detail') | first
}}]
- id: route_decision
type: io.kestra.plugin.core.flow.Switch
description: >-
One branch per decision. The defaults branch exists because a jq program
that errors yields nothing, which would otherwise read as a healthy
cluster: an unreadable detector result fails the run instead of passing
it.
value: "{{ (outputs.classify_findings.values.result | jq('.decision') | first)
?? 'UNREADABLE' }}"
cases:
"NO_FINDINGS":
- id: log_no_findings
type: io.kestra.plugin.core.log.Log
message: "No cancellable instances. Nothing on {{
inputs.allowed_process_definition_ids }} has an open JOB_NO_RETRIES
incident older than {{ inputs.grace_minutes }} minutes whose job is
also sitting at 0 retries."
"OVER_CEILING":
- id: alert_over_ceiling
type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
description: A finding count over the ceiling is its own alert. Nothing is
cancelled, because a spike like this is usually one broken
deployment rather than a backlog worth deleting.
url: "{{ secret('SLACK_WEBHOOK_URL') }}"
payload: |
{
"text": "Camunda stuck instance guard REFUSED to cancel: {{ outputs.classify_findings.values.result | jq('.count') | first }} cancellable instances exceeds the per-run ceiling of {{ inputs.max_cancellations }}. Nothing was cancelled. Check whether a worker pool is down or a bad definition was deployed before raising the ceiling. Execution {{ execution.id }}."
}
- id: log_over_ceiling
type: io.kestra.plugin.core.log.Log
message: "REFUSED: {{ outputs.classify_findings.values.result | jq('.count') |
first }} cancellable instances is over the ceiling of {{
inputs.max_cancellations }}. No instance was cancelled. Candidates:
{{ outputs.classify_findings.values.result | jq('.detail') | first
}}"
"REPORT_ONLY":
- id: log_report_only
type: io.kestra.plugin.core.log.Log
message: "Would cancel {{ outputs.classify_findings.values.result | jq('.count')
| first }} instance(s) and did not, because auto_remediate={{
inputs.auto_remediate }} dry_run={{ inputs.dry_run }}. Candidates:
{{ outputs.classify_findings.values.result | jq('.detail') | first
}}"
"CANCEL":
- id: confirm_cancellation
type: io.kestra.plugin.core.flow.Pause
description: >-
Cancelling ends the instance where it stands. Its variables go, its
token position goes, whatever the process was doing in the real
world is abandoned. Read the candidate list in log_findings before
answering. An instance whose job still has retries belongs in a
retry, not here. The detector already refused those.
onResume:
- id: approved
type: BOOL
defaults: false
description: Answer true to cancel the listed instances. Anything else,
including resuming with no answer, cancels nothing.
- id: check_approval
type: io.kestra.plugin.core.flow.If
description: An unanswered Pause leaves approved null, which is why the
condition defaults it to false rather than testing whether onResume
is defined.
condition: "{{ outputs.confirm_cancellation.onResume.approved ?? false }}"
then:
- id: cancel_stuck_instances
type: io.kestra.plugin.core.flow.Loop
description: >-
One CancelProcessInstance per approved key. concurrencyLimit 1
because these are writes against the same engine. allowFailure
is on deliberately: Camunda refuses to cancel an instance that a
call activity created. One refusal must not stop the re-read
below from reporting the truth about the rest.
concurrencyLimit: 1
values: "{{ outputs.classify_findings.values.result | jq('.keys | tojson') |
first }}"
tasks:
- id: cancel_one_instance
type: io.kestra.plugin.camunda.CancelProcessInstance
description: Terminates one process instance by key through the typed Camunda
plugin.
allowFailure: true
restAddress: "{{ inputs.camunda_rest_address }}"
processInstanceKey: "{{ item.value }}"
- id: settle_secondary_storage
type: io.kestra.plugin.core.flow.Sleep
description: Camunda's v2 search API reads secondary storage, which an exporter
fills asynchronously. Reading it the instant a command returns
can still show the pre-cancel world, so the re-verify waits
first.
duration: PT6S
- id: reverify_cancelled_keys
type: io.kestra.plugin.core.http.Request
description: Ask the engine which of the approved keys are STILL ACTIVE. This
read, not the cancel task going green, is what decides whether
the run succeeded.
uri: "{{ inputs.camunda_rest_address }}/v2/process-instances/search"
method: POST
contentType: application/json
body: |
{
"filter": {
"state": "ACTIVE",
"processInstanceKey": {"$in": {{ outputs.classify_findings.values.result | jq('.keys | tojson') | first }} }
},
"page": {"limit": 1000}
}
- id: read_active_after
type: io.kestra.plugin.core.http.Request
description: Cluster-wide ACTIVE count after the cancellations, for the audit
line to sit beside read_active_before.
uri: "{{ inputs.camunda_rest_address }}/v2/process-instances/search"
method: POST
contentType: application/json
body: |
{
"filter": {"state": "ACTIVE"},
"page": {"limit": 1}
}
- id: guard_reverify
type: io.kestra.plugin.core.flow.If
description: >-
A cancel command Camunda refused still leaves the task green
under allowFailure, so the only trustworthy signal is this
count. Anything but zero fails the run.
condition: "{{ (outputs.reverify_cancelled_keys.body | jq('.page.totalItems') |
first) != 0 }}"
then:
- id: fail_reverify
type: io.kestra.plugin.core.execution.Fail
errorMessage: >-
Re-verify failed. {{ outputs.reverify_cancelled_keys.body |
jq('.page.totalItems') | first }} of {{
outputs.classify_findings.values.result | jq('.count') |
first }} approved instance(s) are still ACTIVE after
CancelProcessInstance returned: {{
outputs.reverify_cancelled_keys.body | jq('[.items[] |
((.processInstanceKey | tostring) + " (" +
(.processDefinitionId | tostring) + ", parent=" +
((.parentProcessInstanceKey | tostring) // "none") + ")")] |
join(", ")') | first }}. Camunda refuses to cancel an
instance a call activity created, cancel the root instance
instead.
else:
- id: log_cancellation_verified
type: io.kestra.plugin.core.log.Log
message: >-
Cancelled and verified {{
outputs.classify_findings.values.result | jq('.count') |
first }} instance(s). None of the approved keys is ACTIVE
any more. Cluster ACTIVE instances {{
outputs.read_active_before.body | jq('.page.totalItems') |
first }} before, {{ outputs.read_active_after.body |
jq('.page.totalItems') | first }} after. Keys: {{
outputs.classify_findings.values.result | jq('.keys |
join(", ")') | first }}
else:
- id: log_cancellation_declined
type: io.kestra.plugin.core.log.Log
message: "Cancellation declined at the approval gate. {{
outputs.classify_findings.values.result | jq('.count') | first
}} instance(s) left exactly as they were: {{
outputs.classify_findings.values.result | jq('.detail') | first
}}"
defaults:
- id: fail_unreadable_detector
type: io.kestra.plugin.core.execution.Fail
errorMessage: "The detector produced no readable decision, so this run cannot
tell a clean cluster from a stuck one. Raw result: {{
outputs.classify_findings.values.result }}"
- id: log_audit_complete
type: io.kestra.plugin.core.log.Log
description: Runs on every path so the execution history doubles as a record of
how many instances were parked on an incident over time.
message: "Stuck instance guard complete. decision={{
outputs.classify_findings.values.result | jq('.decision') | first }}
cancellable={{ outputs.classify_findings.values.result | jq('.count') |
first }} grace_minutes={{ inputs.grace_minutes }} ceiling={{
inputs.max_cancellations }} auto_remediate={{ inputs.auto_remediate }}
dry_run={{ inputs.dry_run }} allowlist={{
inputs.allowed_process_definition_ids }}"
errors:
- id: alert_guard_failure
type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
description: Fires when the run itself fails, which includes the re-verify
refusing to accept a cancellation that did not land. A failed sweep must
never read as a clean cluster.
url: "{{ secret('SLACK_WEBHOOK_URL') }}"
payload: |
{
"text": "Camunda stuck instance guard FAILED in execution {{ execution.id }} against {{ inputs.camunda_rest_address }}. Either the cluster could not be read or an approved cancellation did not land and the re-verify refused it. Check the execution before assuming no instance is parked on an incident."
}