Couchbase Cluster Node Rebalance Sentinel Watchdog

Couchbase redistributes data, indexes, and query processing across nodes during rebalance, and the cluster's REST API exposes the current status at /pools/default/rebalanceProgress, but nothing reads that endpoint unless you build it. The plugin-couchbase Kestra plugin only runs N1QL queries and query-triggered flows, it has no cluster-admin or rebalance task, so this blueprint calls the REST API directly with the core HTTP task, parses the status field with jq, and reports idle, in progress, or unreachable.

How it works

1. fetch_rebalance_status (io.kestra.plugin.core.http.Request) issues a GET against couchbase_url with HTTP Basic Auth (couchbase_username / COUCHBASE_PASSWORD), options.allowFailed: true so a non-2xx response (for example a wrong password) is returned as a normal response instead of failing the task, and allowFailure: true so a total connection failure still lets the flow continue.
2. extract_status_field (io.kestra.plugin.core.debug.Return) reads outputs.fetch_rebalance_status.body with the jq Pebble filter to pull out the status field, defaulting to an empty string when the body is missing entirely or has no status key.
3. classify_outcome (io.kestra.plugin.core.debug.Return) maps that field to REBALANCING (running), IDLE (none), or UNREACHABLE (anything else, including empty).
4. route_by_outcome (io.kestra.plugin.core.flow.Switch) branches on the outcome: IDLE logs a quiet confirmation; REBALANCING logs and, when notify_on_rebalance is true, posts a Slack message with the node count and lowest per-node progress parsed from the same body; UNREACHABLE always pages Slack; defaults logs the raw outcome.
5. The errors block pages Slack separately if the flow itself fails outright, and the couchbase_rebalance_schedule trigger polls hourly, shipped disabled: true until you point it at a real endpoint.

What you get

• A scheduled, authenticated read of Couchbase's /pools/default/rebalanceProgress endpoint with zero custom client code.
• A three-way status output (IDLE / REBALANCING / UNREACHABLE) you can chart or gate other flows on, plus an optional in-progress Slack notice and an always-on unreachable page.

Who it's for

• Database reliability and platform teams running self-managed Couchbase clusters with no first-class orchestrator-side rebalance check today.
• On-call engineers who want a signal when a rebalance is active or the admin API becomes unreachable.

Why orchestrate this with Kestra

A cron job that curls an admin endpoint and exits is invisible the moment it succeeds. This flow keeps every check in the execution history, turns the result into a typed output other flows can depend on, and gives you a Switch branch to grow into real remediation (paging the on-call DBA, opening a ticket) without rewriting the check itself.

Prerequisites

• A running Couchbase cluster with its REST admin API reachable from the Kestra Worker on port 8091 and already initialized with an admin user (a fresh container answers nothing useful until initialized). For local testing, start the community edition: docker run -d --name couchbase -p 8091-8097:8091-8097 -p 11210-11211:11210-11211 couchbase/server:community, then initialize it: docker exec couchbase couchbase-cli cluster-init -c 127.0.0.1 --cluster-username Administrator --cluster-password <your-password> --services data --cluster-ramsize 1024, then connect it to your Kestra worker's Docker network so couchbase_url resolves: docker network connect YOUR_KESTRA_NETWORK couchbase (run docker network ls to find the real network name).
• A Slack incoming webhook for the rebalance and unreachable alerts.

Secrets

• COUCHBASE_PASSWORD: password for the couchbase_username administrator account, used for HTTP Basic Auth against the REST API.
• SLACK_WEBHOOK_URL: Slack incoming webhook used for the rebalance notice, the unreachable alert, and the flow-failure alert.
• OSS note: if your Kestra instance reads secrets from environment variables instead of a secrets backend, set SECRET_COUCHBASE_PASSWORD and SECRET_SLACK_WEBHOOK_URL as base64, for example echo -n "<your-password>" | base64 and echo -n "<your-webhook-url>" | base64.

Inputs

• couchbase_url (STRING, default http://couchbase:8091/pools/default/rebalanceProgress): full URL to the rebalanceProgress endpoint.
• couchbase_username (STRING, default Administrator): admin username for Basic Auth.
• notify_on_rebalance (BOOLEAN, default true): post a Slack message when a rebalance is found running, in addition to the always-on unreachable alert.

Quick start

1. Add the COUCHBASE_PASSWORD and SLACK_WEBHOOK_URL secrets.
2. Set couchbase_url to your cluster's rebalanceProgress endpoint, e.g. http://localhost:8091/pools/default/rebalanceProgress.
3. Run the flow once on a freshly initialized single node and confirm status reads IDLE.
4. Enable couchbase_rebalance_schedule.

Outputs

• status (STRING): the classified outcome, {{ outputs.classify_outcome.value }}, also exposed as the flow-level output status.
• {{ outputs.fetch_rebalance_status.body }} and {{ outputs.fetch_rebalance_status.code }}: raw response body and HTTP status code of the check, when the request completed.
• {{ outputs.extract_status_field.value }}: the raw status field (running, none, or empty string) before classification.

Pitfalls

• A fresh, uninitialized container answers the REST API with neither a useful status nor valid credentials; run cluster-init first or every check reports UNREACHABLE.
• IDLE is the only outcome you can test on a single node, since a one-node cluster has nothing to rebalance. Testing REBALANCING needs a real rebalance on a multi-node cluster (add or remove a node and rebalance while the flow runs). Testing UNREACHABLE is easiest by setting a wrong COUCHBASE_PASSWORD or stopping the container.
• This flow does not detect a stalled or stuck rebalance and does not enforce any duration budget: the rebalanceProgress API returns no start time or elapsed duration, only the current per-node progress fraction, so "has this been running too long" cannot be computed from this endpoint alone.
• An hourly poll can miss a rebalance that starts and finishes between two runs; it will simply never observe REBALANCING for that event.
• Per-node progress is a fraction; the documented examples show values like 0.11 while rebalancing, but the valid numeric range is not stated in the cited documentation page, so treat it as an indicative, not a calibrated, percentage.

How to extend

• Store the per-node progress numbers in the KV store and compare successive runs to build your own stall detector (for example, flag it if progress has not increased since the last check), as a future idea only, since the API itself gives no duration signal today.
• Push status into a KV store to alert only on state transitions instead of every run.

Links

• Couchbase Getting Rebalance Progress REST API
• couchbase-cli cluster-init reference
• Kestra Switch task