Audit Couchbase GSI Secondary Indexes and Detect Offline States

Couchbase Global Secondary Indexes (GSI) provide low-latency lookups and filtering for N1QL and SQL++ query workloads across distributed document collections. During cluster rebalances, node failovers, high mutation volume, or memory quota exhaustion, GSI indexes can transition into building, deferred, offline, or error states in Couchbase's internal system:indexes catalog. When indexes drop offline, queries that depend on them either fail outright with query errors or fall back to brute-force primary key scans, multiplying CPU utilization and causing cascading latency spikes across application tiers.

This blueprint implements an automated Day 2 Operations sentinel that queries Couchbase on a recurring schedule. It executes a N1QL query against the internal system:indexes catalog to detect any indexes in non-online states, evaluates degraded index counts against operational thresholds, and dispatches real-time Slack alerts to database administrators with the exact bucket, index name, state, and remediation query before query timeouts affect live traffic.

How it works

1. periodic_index_audit (io.kestra.plugin.core.trigger.Schedule) triggers the audit pass every 4 hours (shipped disabled for safe initial onboarding).
2. probe_unhealthy_indexes (io.kestra.plugin.couchbase.Query) connects to the Couchbase cluster, executing a N1QL query against system:indexes to select all indexes whose state is not online.
3. evaluate_index_health (io.kestra.plugin.core.flow.Switch) inspects outputs.probe_unhealthy_indexes.size to classify cluster index health into HEALTHY or INDEX_DEGRADATION_DETECTED.
4. When within safe operating limits (HEALTHY), log_indexes_healthy records that 0 degraded indexes were detected.
5. When degraded indexes are found (INDEX_DEGRADATION_DETECTED), alert_index_degradation sends an operational notification to Slack via io.kestra.plugin.slack.notifications.SlackIncomingWebhook with index name, bucket, and state, while log_remediation_guidance logs actionable BUILD INDEX guidance.
6. log_audit_completion writes a final audit recap for execution tracking and lineage.
7. The flow-level errors block alerts Slack immediately if Couchbase authentication or network connectivity fails.

What you get

• Early detection of stalled, building, or offline GSI indexes before query timeouts impact production APIs.
• Cluster-wide visibility across all buckets, scopes, and collections in a single N1QL query.
• Automated Slack incident notifications with exact index names, bucket mappings, and degradation states.
• Actionable remediation logs with specific N1QL commands to build or rebuild affected indexes.
• Flow-level error alerting safeguarding audit pipeline reliability.

Who it's for

• Database Administrators (DBAs) and Site Reliability Engineers managing Couchbase Server clusters.
• Platform Engineers maintaining distributed NoSQL document databases in production.
• Backend Engineers operating high-throughput microservices querying Couchbase via N1QL/SQL++.

Why orchestrate this with Kestra

Monitoring Couchbase system catalogs with detached cron jobs or external scripts requires managing Couchbase C-SDK/Java client dependencies, credential rotation, error handling, and alerting integrations in fragile custom scripts. Kestra provides a declarative workflow engine with native Couchbase N1QL query execution, centralized secret management, conditional branching, execution lineage, and native Slack integrations without maintaining separate monitoring infrastructure.

Prerequisites

• A running Couchbase Server cluster (Community Edition v7.x or Enterprise Edition).
• Administrative credentials with Query and Index reader privileges.
• A Slack Incoming Webhook URL configured in Kestra secrets.

Local Docker Container Setup

To run Couchbase Community Edition locally for development and testing without host port conflicts:

docker run -d --name couchbase-dev -p 8091-8096:8091-8096 -p 11210:11210 \
  couchbase/server:community

If Kestra is running inside a Docker network, connect the Couchbase container to Kestra's network:

docker network ls
docker network connect YOUR_KESTRA_NETWORK couchbase-dev

Secrets

• COUCHBASE_PASSWORD: Password for the Couchbase administrative user.
• SLACK_WEBHOOK: Incoming Webhook URL for Slack alerting.

Note for Kestra Open-Source (OSS): Secrets are configured as base64-encoded environment variables formatted as SECRET_<NAME> (e.g. SECRET_COUCHBASE_PASSWORD).

Inputs

Name	Type	Default	Allowed Values	Description
connection_string	STRING	"couchbase://localhost"	Valid Couchbase URI	Cluster connection endpoint
username	STRING	"Administrator"	String identifier	Cluster administrative username
target_bucket	STRING	"ALL"	Bucket name or "ALL"	Target bucket filter
remediation_mode	SELECT	"ALERT_ONLY"	"ALERT_ONLY", "LOG_GUIDANCE"	Action mode on index degradation

Outputs

• {{ outputs.probe_unhealthy_indexes.rows }}: List of maps containing name, bucket_name, and state for every non-online index.
• {{ outputs.probe_unhealthy_indexes.size }}: Total count of non-online secondary indexes detected.

Quick start

1. Configure COUCHBASE_PASSWORD and SLACK_WEBHOOK in your Kestra namespace secrets.
2. Import this blueprint into your Kestra instance.
3. Run a manual execution with connection_string pointing to your cluster to verify connectivity and index catalog query.
4. Enable the periodic_index_audit schedule trigger to run automated recurring audits.

How to extend

• Add automated index rebuilding by invoking a downstream io.kestra.plugin.couchbase.Query task executing BUILD INDEX when indexes are found in deferred state.
• Integrate with PagerDuty or Opsgenie in the INDEX_DEGRADATION_DETECTED branch for critical customer-facing service indexes.
• Chain an automated index RAM quota check subflow using io.kestra.plugin.core.flow.Subflow to evaluate whether Index service memory limits need expansion.

Pitfalls

• Index Building vs Stalled States: Immediately after creating large indexes, state naturally reports building while initial indexing completes. Ensure audit thresholds or schedules do not falsely alert during planned maintenance windows.
• Port Accessibility: Couchbase requires port 8091 (REST API / Cluster Manager), port 8093 (Query Service), and port 11210 (Memcached/Data Service) accessible between the Kestra worker and Couchbase nodes.
• Keyspace Identification: In Couchbase 7.0+, indexes can belong to custom scopes and collections. The query uses IFMISSING(bucket_id, keyspace_id) to resolve keyspace names across both legacy and modern collection-based buckets.

Links

• Kestra Couchbase Plugin Documentation
• Couchbase System Catalogs Documentation
• Couchbase GSI Index Lifecycle Guide
• Kestra Slack Notification Plugin