Weaviate Class Population Canary Gate

A Weaviate class that gets accidentally emptied - a bad Delete call with too broad a filter, an ingest pipeline that silently stopped, a collection dropped and never rebuilt - still answers every query without error. It just returns nothing useful, and the first person to notice is usually whoever is on the receiving end of a RAG pipeline that suddenly has no grounding data. This blueprint runs Weaviate's own GraphQL Aggregate meta { count } query against a named class on a schedule and alerts Slack the moment the count drops below a minimum you set.

How it works

1. periodic_population_check (io.kestra.plugin.core.trigger.Schedule) runs every 30 minutes. Shipped disabled so you can validate a manual run first.
2. check_class_population (io.kestra.plugin.weaviate.Query, fetchType: FETCH_ONE) runs { Aggregate { class_name { meta { count } } } } against the Weaviate GraphQL endpoint, landing the result at row.meta.count.
3. evaluate_population (io.kestra.plugin.core.flow.If) branches on row.meta.count >= min_object_count.
   • then (HEALTHY): log_population_healthy records a clean check.
   • else (AT_RISK): alert_population_at_risk reports the measured count against the minimum. There is deliberately no remediation action here - see Pitfalls for why.
4. log_audit_complete always runs last, printing the headline numbers regardless of which branch fired.
5. The errors block alerts Slack separately if the flow itself fails outright - an unreachable instance, a bad API key, or class_name not existing at all, since Weaviate's GraphQL layer errors on an undefined class instead of returning a zero count.

What you get

• A direct count from Weaviate's own GraphQL Aggregate query, not an estimate from a dashboard or a downstream symptom report.
• A distinct failure path (the errors block) for a class that does not exist at all, so that case is never silently read as "zero objects, below threshold" - it is a different, more severe condition.
• A one-number threshold (min_object_count) that works equally as an always-populated canary (default 1) or a partial-data-loss detector on a class expected to hold far more.
• A final status log every run, risk or not, so the execution history doubles as a population trend for the class.

Who it's for

• Platform and RAG infrastructure teams running Weaviate who need an early signal when a class that feeds a retrieval pipeline silently loses its data.
• Teams who have been burned by a Delete call with a filter that matched more than intended and want a scheduled check that would have caught it within the hour instead of the next incident review.
• Anyone evaluating the Weaviate plugin who wants a worked example of Query running a raw GraphQL Aggregate request, not just Get-style data retrieval.

Why orchestrate this with Kestra

An Aggregate query run by hand answers one moment in time; it does not decide a cadence, compare the result against a threshold, or notify anyone. Kestra supplies the schedule, the threshold comparison as a first-class branch, and an execution history that shows exactly when a class's population dropped and by how much.

Prerequisites

• A reachable Weaviate instance with class_name already created and expected to be populated.
• An API key with permission to run GraphQL queries, if the instance has authentication enabled.
• A Slack incoming webhook for alerts.

Local testing: docker run -d --name weaviate-canary -p 8081:8080 -p 50052:50051 -e QUERY_DEFAULTS_LIMIT=25 -e AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true -e PERSISTENCE_DATA_PATH=/var/lib/weaviate -e DEFAULT_VECTORIZER_MODULE=none cr.weaviate.io/semitechnologies/weaviate:1.25.0 docker network connect YOUR_KESTRA_NETWORK weaviate-canary (check existing networks first with docker network ls; only needed if the Kestra Worker runs in a separate Docker network than this container; set weaviate_url to http://weaviate-canary:8080 from inside that network, or http://localhost:8081 from the host). Host port 8081 is used here specifically so it never collides with Kestra's own UI on 8080. With AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true, WEAVIATE_API_KEY can be set to any placeholder value - the instance does not check it.

Secrets

• WEAVIATE_API_KEY: API key used by check_class_population's apiKey property.
• SLACK_WEBHOOK_URL: Slack incoming webhook used by alert_population_at_risk and the errors block.
• In Kestra OSS (no Enterprise secrets backend), secrets are supplied as environment variables prefixed SECRET_, base64-encoded, and read back in flows with {{ secret('NAME') }} - for example SECRET_WEAVIATE_API_KEY=$(echo -n '<your-local-key>' | base64). This keeps credentials out of the flow YAML but, per Kestra's own documentation, offers no encryption at rest or access control beyond the host environment; use the Enterprise secrets backend for stronger guarantees.

Inputs

• weaviate_url (STRING, default http://localhost:8081): Weaviate instance URL.
• class_name (STRING, default Article): class (collection) audited.
• min_object_count (INT, default 1): count floor before the class counts as AT_RISK.

Outputs

• outputs.check_class_population.row.meta.count: the measured object count, every successful run.

Quick start

1. Start Weaviate locally with the command above and create an Article class with a handful of objects (via SchemaCreate and BatchCreate, or the Weaviate console).
2. Add the WEAVIATE_API_KEY and SLACK_WEBHOOK_URL secrets.
3. Run the flow manually with the defaults and confirm log_population_healthy fires with the correct count.
4. Raise min_object_count above the actual count and re-run to confirm alert_population_at_risk fires.
5. Point class_name at a class that does not exist and re-run to confirm the errors block fires instead of either branch.
6. Enable periodic_population_check once you trust the check.

How to extend

• Loop over several class_name values with io.kestra.plugin.core.flow.Loop to run the same canary across every class a RAG pipeline depends on in one flow.
• Add a second Aggregate query filtered with a where clause on a recency field (if the class has one) to check not just total population but how many objects were added in the last run window.
• Store row.meta.count to the KV store after each run and compare against the previous value, so a slow leak (not just a sudden wipe) also trips an alert.
• Chain a downstream Subflow call to a reingestion pipeline on the AT_RISK branch, if one exists, instead of leaving remediation as alert-only.

Pitfalls

• There is no remediation task in this plugin for a population drop. Verified from the plugin's own source: only BatchCreate, Delete, Query, SchemaCreate exist under io.kestra.plugin.weaviate - nothing to reindex or restore lost objects, which is why the AT_RISK branch only alerts, unlike this repo's other governance gates that gate a remediation action behind auto_remediate/dry_run.
• A missing class is not the same as a zero count. Aggregate on an undefined class_name returns a GraphQL error, which this plugin's Query task turns into a thrown exception - caught by the errors block, not by evaluate_population. A typo'd class_name reads as a flow failure, not as AT_RISK.
• FETCH_ONE's flattening means row is keyed by the query's structure, not a fixed path. outputs.check_class_population.row.meta.count only works because the GraphQL query asks for Aggregate { class_name { meta { count } } } } specifically - verified directly from the plugin's extractRow logic, which takes the first class's first row-map, not a documented, stable output field name.
• class_name is spliced directly into the GraphQL query text. There is no bind-parameter mechanism for a GraphQL type name in this query; treat class_name as a trusted input, not one sourced from unescaped external user input.
• This blueprint is UNTESTED against a live Weaviate instance. The Aggregate/meta { count } query shape is taken verbatim from Weaviate's own GraphQL API documentation, and the Query task's properties (url, apiKey, query, fetchType) and response-flattening behavior are taken verbatim from the plugin's source on main - but no Docker container or Kestra engine was run to execute this flow end to end.

Links

• plugin-weaviate documentation
• plugin-weaviate source (Query.java)
• Weaviate GraphQL Aggregate API documentation
• Kestra secrets (OSS vs Enterprise)