Monitor Pinecone Vector Counts and Alert on Silent Data Loss

A RAG corpus can shrink without any error being raised: a purge targets the wrong namespace, an ingestion job silently stops, a migration drops half the vectors. Retrieval keeps returning results, just worse ones. This blueprint watches the number that gives it away. On a schedule, io.kestra.plugin.pinecone.DescribeIndexStats reads the index and an If task compares totalVectorCount against a minimum threshold. Below it, Slack gets paged and io.kestra.plugin.core.execution.Fail marks the execution failed so the breach shows up in Kestra's UI, metrics, and alerting, not just in a chat channel.

How it works

1. check_stats (io.kestra.plugin.pinecone.DescribeIndexStats) reads the index named by index_name and outputs totalVectorCount, the number of vectors across all namespaces.
2. evaluate_threshold (io.kestra.plugin.core.flow.If) compares that count against the min_vector_count input.
3. On a breach, alert_low_count pages Slack with the observed count and the threshold, then fail_execution fails the run with an explicit error message.
4. When healthy, log_healthy writes the count to the logs, turning the execution history into a capacity timeline.
5. The errors block posts a Slack message whenever the flow ends in failure, covering both a threshold breach and a broken stats call.
6. A disabled-by-default Schedule trigger runs the check every six hours.

What you get

• Detection of silent corpus shrinkage within hours instead of after user complaints.
• A breach that is both a Slack page and a failed execution, so it feeds any alerting built on Kestra's execution states.
• A per-check log of the vector count, useful for spotting slow leaks as well as cliffs.
• Two inputs and no code to adapt the monitor to any index and corpus size.

Who it's for

• Teams running RAG in production who treat the vector index as a tier-one dependency.
• Platform engineers who already alert on failed Kestra executions and want vector health in the same pipeline.
• Anyone who has run the retention purge blueprint and wants a backstop against purging the wrong namespace.

Why orchestrate this with Kestra

Pinecone reports the stats; someone still has to look at them on a schedule, remember what normal is, and raise a hand when the number is wrong. Kestra runs the readback four times a day, encodes the threshold as an input under version control, and turns a breach into a failed execution with an alert. The monitor itself is monitored: if the stats call breaks, the errors block fires instead of the check silently stopping.

Prerequisites

• A Pinecone index carrying a corpus whose expected size you know roughly.
• A min_vector_count value set just below that size, leaving headroom for normal churn.
• A Slack incoming webhook for alerts.

Secrets

• PINECONE_API_KEY: Pinecone API key with read permission on the index.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the PINECONE_API_KEY and SLACK_WEBHOOK_URL secrets to your Kestra namespace.
2. Execute the flow once with min_vector_count above your real count and confirm the Slack alert and failed execution arrive.
3. Set the threshold below the real corpus size and set disabled: false on the every_six_hours trigger.

How to extend

• Alert on a specific namespace by comparing an entry of the namespaceCounts output map instead of the total.
• Add an upper-bound check to catch runaway ingestion doubling the corpus.
• Post the healthy count to a metrics system on the else branch to build a dashboard of corpus size over time.
• Trigger the embeddings upsert blueprint automatically on a breach to rehydrate the index from the latest export.

Links

• Pinecone plugin documentation
• Pinecone documentation