Cancel runaway Couchbase N1QL queries that starve the query service

One runaway N1QL request can starve the whole Couchbase query service. An unindexed full-keyspace scan or a cartesian join pins a query node on CPU and memory. Every other query on that node slows down or stalls behind it. Couchbase has no circuit breaker for this, so the request runs until it finishes, times out or someone cancels it by hand. This blueprint reads system:active_requests, finds requests whose elapsed time is over a threshold, cancels each one by requestId behind a safe gate, then re-reads system:active_requests and fails the run if anything is still over the threshold.

This blueprint was created by zkasuran.

Safety

Cancelling a request is disruptive to whoever launched it. The request stops where it is and its partial work is discarded, so a half-finished write-shaped statement leaves nothing behind and a read returns nothing. The controls are in the flow, not only in this text.

• Read-only by default. dry_run defaults to true and auto_remediate to false, so a plain run only reports. A cancel runs only when auto_remediate is true, dry_run is false and a human answers the Pause with approved: true. An unanswered resume reads as not approved.
• The flow never cancels its own request. The detector excludes any statement that reads system:active_requests, which is exactly what the detect query, the cancel statements and the re-read all do, so none of them can ever appear in the target set. A monitoring query against the same keyspace is left alone for the same reason.
• The threshold defaults to 300 seconds. A short query is never a target. Raise or lower elapsed_threshold_seconds to match the slowest query the workload runs on purpose.
• The cancel loop runs with concurrencyLimit: 1, so cancels are serialised rather than racing.
• A request that finishes on its own between detect and cancel is a no-op, not an error. The cancel is marked allowFailure, so a cancel of a request that already left system:active_requests does not abort the batch, so the re-read is the single source of truth on whether the service is clear.

How it works

1. detect (io.kestra.plugin.couchbase.Query) runs SELECT ... FROM system:active_requests WHERE STR_TO_DURATION(elapsedTime) > <threshold> AND statement NOT LIKE '%active_requests%' with fetchType: FETCH, so every long-running request that is not a monitoring query comes back in outputs.detect.rows.
2. plan (io.kestra.plugin.core.output.OutputValues) turns the rows into the count, the list of requestId values to cancel and a label for the alert.
3. report (io.kestra.plugin.core.log.Log) logs the count and the full detail.
4. on_findings (io.kestra.plugin.core.flow.If) posts a Slack findings alert through io.kestra.plugin.slack.notifications.SlackIncomingWebhook when the count is above zero. This fires on a dry run too.
5. remediate (io.kestra.plugin.core.flow.If) runs only when there is a runaway and auto_remediate is true and dry_run is false. approval_hold (io.kestra.plugin.core.flow.Pause) waits for a human. On approval cancel_runaway (io.kestra.plugin.core.flow.Loop, concurrencyLimit: 1) issues one DELETE FROM system:active_requests WHERE requestId = <id> per request, the documented way to cancel a running N1QL request.
6. reverify (io.kestra.plugin.core.flow.LoopUntil) re-reads system:active_requests on an interval, bounded by poll_max_duration. A cancel is not always instant, so the loop ends when no request is over the threshold and fails the run if the window closes first. confirm_cancelled logs the success. On a declined approval the flow logs it and cancels nothing. The errors block posts a Slack alert on any failure: a bad credential, an unreachable cluster or a request still over the threshold after the cancel.

What you get

• A scheduled read-only check that names the requests starving the query service.
• A conservative default threshold so only a genuinely stuck request is ever a target.
• A real cancel through DELETE FROM system:active_requests, gated on a dry run, an explicit flag and a human approval.
• A re-read that fails the run when a request is still over the threshold, so a cancel that did not land never passes as healthy.
• A detector that excludes its own request by construction, proven by a run with the threshold set to zero.
• Slack alerts on detection and on failure.

Who it's for

• Couchbase operators who need a runaway N1QL request stopped before it takes the query node down.
• Platform teams running shared query nodes where one bad statement hurts every tenant.
• On-call engineers who want the stuck request named and cancellable from one place.

Why orchestrate this with Kestra

Couchbase does not stop a request for using too much time. It runs until it finishes or hits its own timeout. Until then it holds CPU and memory on the query node. Finding it needs a read of system:active_requests. Stopping it needs a cancel that must never hit the wrong request. Kestra runs the read on a schedule, keeps the cancel behind an approval, re-reads the cluster to confirm the service is clear, then routes findings and failures to Slack. Every run is an execution you can replay with the exact requests it acted on.

Tested end to end

Verified on Kestra 2.0.5 against a real Couchbase cluster (couchbase/server:community) with the data, index and query services enabled, confirmed before and after with direct N1QL queries against system:active_requests.

• Clean cluster, no long request: detect found 0, no alert fired, nothing was cancelled.
• Degraded cluster with one seeded cartesian runaway at dry_run=true: detect found exactly that one request, a findings alert posted, nothing was cancelled. A direct N1QL read confirmed the request still running afterward.
• Never cancels itself: with elapsed_threshold_seconds=0 and no runaway, every active request qualified on time yet detect returned 0, because the only active request was the detector itself and it is excluded by the statement filter.
• dry_run=true with auto_remediate=true: still no cancel. The dry-run gate held.
• dry_run=false with auto_remediate=true, approved: the request was cancelled, the re-read returned 0 and a direct N1QL read confirmed the request was gone. The before read showed the real requestId and a climbing elapsedTime.
• Approval declined: the flow logged the decline and the request was still running afterward.
• Re-verify failure: a second runaway launched during the Pause was still over the threshold after the approved cancel of the first, the re-read saw it and the run failed instead of reporting success.
• Failure path: a wrong password failed detect and the errors block posted a Slack alert.

Prerequisites

Couchbase needs the query and index services enabled, otherwise system:active_requests is not reachable through N1QL. The user needs privileges to read system:active_requests and to cancel requests, which the full administrator role grants.

Secrets

• COUCHBASE_PASSWORD: password for the Couchbase user. It is the default for the password input.
• SLACK_WEBHOOK_URL: incoming webhook used by the findings alert and the error alert.

Inputs

• connection_string: Couchbase connection string, for example couchbase://host1,host2.
• username: Couchbase user allowed to read system:active_requests and cancel requests.
• password: user password, defaulting to the COUCHBASE_PASSWORD secret.
• elapsed_threshold_seconds: a request is a target when its elapsed time is over this many seconds. Defaults to 300.
• dry_run: report only when true. Defaults to true.
• auto_remediate: cancel the requests when true and dry_run is false. Defaults to false.
• poll_max_duration: upper bound on the re-read poll. Defaults to PT30S.

Quick start

1. Start Couchbase with the data, index and query services, then create a bucket.
2. Launch a long query that does a full keyspace scan or a cartesian join so there is something to find.
3. Set the COUCHBASE_PASSWORD and SLACK_WEBHOOK_URL secrets in Kestra.
4. Run with dry_run true to list the requests over the threshold.
5. Re-run with dry_run false and auto_remediate true, then answer the Pause with approved: true.
6. Enable the schedule for an hourly safe-mode check.

How to extend

• Lower elapsed_threshold_seconds on a workload with a tight query budget. Raise it where long analytical queries are normal.
• Add the request node to the alert and group by it to see which query node is under pressure.
• Keep a KV cursor of the requestId values seen over the threshold so a request is only cancelled after it has been slow across two runs.
• Cancel only requests from a given userAgent or clientContextID by adding the column to the detect filter.

Links

• Couchbase plugin documentation
• Couchbase system catalog keyspaces
• Monitoring and cancelling active requests