Detect and build Couchbase indexes left deferred after defer_build

Couchbase indexes created with defer_build sit in a deferred state until a BUILD INDEX runs. If that step is skipped the index never comes online. N1QL quietly falls back to a primary scan and queries slow down with no error to point at. This blueprint finds those indexes, builds them behind a safe gate, then re-reads system:indexes to prove the build finished.

This blueprint was created by zkasuran.

Safety

Read-only by default. The dry_run input defaults to true and auto_remediate defaults to false, so a plain run only reports. BUILD INDEX runs only when auto_remediate is true and dry_run is false. The re-verify step fails the run if any targeted index is still not online when the poll window closes, so a half-built index never passes as healthy.

How it works

1. detect (io.kestra.plugin.couchbase.Query) runs SELECT name, state FROM system:indexes WHERE keyspace_id = <bucket> AND state != 'online' with fetchType: FETCH, so every non-online index on the bucket default collection comes back in outputs.detect.rows.
2. report (io.kestra.plugin.core.log.Log) logs the count and the detail.
3. on_findings (io.kestra.plugin.core.flow.If) posts a Slack findings alert through io.kestra.plugin.slack.notifications.SlackIncomingWebhook when the count is above zero. This fires on any run that finds something, a dry run included.
4. remediate (io.kestra.plugin.core.flow.If) runs only when findings exist and auto_remediate is true and dry_run is false. build_index issues one BUILD INDEX ON <bucket>(name, ...) covering every detected index. Couchbase builds deferred indexes listed together in a single scan, which is why the names are batched into one statement.
5. wait_until_online (io.kestra.plugin.core.flow.LoopUntil) re-reads system:indexes on an interval, bounded by poll_max_duration. BUILD INDEX is asynchronous, so a single re-read right after it can still see building. The loop ends when no targeted index is non-online and fails the run if the window closes first.
6. confirm_online logs the success. The errors block posts a Slack alert on any failure: a bad credential, an unreachable cluster or a build that does not finish in time.

What you get

• A scheduled check that catches indexes stuck in deferred or pending after a bulk defer_build.
• A safe default that reports without changing anything.
• A real BUILD INDEX remediation that batches every deferred index into one build.
• A re-verify step that proves the fix landed instead of assuming it.
• Slack alerts on detection and on failure.

Who it's for

• Couchbase operators and platform teams running N1QL workloads.
• Teams that create indexes with defer_build in bulk and need the build step confirmed.
• On-call engineers who want silent primary-scan fallbacks surfaced early.

Why orchestrate this with Kestra

Couchbase has no built-in watchdog for an index that was created but never built. A deferred index stays invisible until queries slow down under a primary scan. Kestra runs the check on a schedule, keeps the write behind an explicit gate, re-reads the cluster to confirm the build, then routes findings and failures to Slack. Every run is a logged execution you can replay.

Tested end to end

Verified on Kestra 2.0.5 against a real Couchbase cluster (couchbase/server:community) with the data, index and query services enabled.

• Clean bucket, every index online: detect found 0, no alert fired, nothing was built.
• Degraded bucket with two deferred GSIs at dry_run=true: detect found 2, a findings alert posted, nothing was built. cbq confirmed both indexes still deferred afterward.
• dry_run=true with auto_remediate=true: still no build. The dry-run gate held.
• dry_run=false with auto_remediate=true: BUILD INDEX ran, the poll waited for the build and the re-verify confirmed both indexes online. cbq confirmed the before state (2 deferred) and the after state (0 deferred).
• Failure path: a wrong password made detect fail and the errors block posted a Slack alert.
• Re-verify failure: against a bucket of roughly 200000 documents with poll_max_duration set to one second, the build could not finish in the window, wait_until_online reached its max duration, the run failed and the errors alert fired. The background build then completed, which is the asynchronous behavior the poll exists to handle.

Prerequisites

Couchbase must have the query and index services enabled at cluster init. Without them system:indexes and BUILD INDEX both fail. The user needs privileges to read system:indexes and to build indexes on the target bucket.

Secrets

• COUCHBASE_PASSWORD: password for the Couchbase user. It is the default for the password input.
• SLACK_WEBHOOK_URL: incoming webhook used by the findings alert and the error alert.

Inputs

• connection_string: Couchbase connection string, for example couchbase://host1,host2.
• username: Couchbase user with query and index privileges.
• bucket: the bucket whose default-collection indexes are inspected.
• password: user password, defaulting to the COUCHBASE_PASSWORD secret.
• dry_run: report only when true. Defaults to true.
• auto_remediate: build the indexes when true and dry_run is false. Defaults to false.
• poll_max_duration: upper bound on the online re-verify poll. Defaults to PT2M.

Quick start

1. Start Couchbase with the data, index and query services, then create a bucket.
2. Create at least one index with the defer_build option set to true so there is something to find.
3. Set the COUCHBASE_PASSWORD and SLACK_WEBHOOK_URL secrets in Kestra.
4. Run with dry_run true to list the deferred indexes.
5. Re-run with dry_run false and auto_remediate true to build them.
6. Enable the schedule for a daily safe-mode check.

How to extend

• Target a named scope and collection by changing the system:indexes filter and the BUILD INDEX keyspace path.
• Add a Pause before build_index for a human approval gate on production.
• Widen detection across every keyspace by grouping on keyspace_id and building per keyspace.
• Lower poll_max_duration for small indexes or raise it for large ones.

Links

• Couchbase plugin documentation
• Couchbase BUILD INDEX reference
• Couchbase system catalog keyspaces