Enforce Retention on a Pinecone Namespace with a Weekly Purge

Experiment vectors outlive experiments. Namespaces used for A/B tests, one-off demos, or model comparisons keep serving nearest-neighbor results long after anyone remembers loading them, polluting retrieval and inflating storage. This blueprint enforces a retention policy in one call: io.kestra.plugin.pinecone.DeleteVectors with deleteAll: true clears the namespace passed as input, io.kestra.plugin.pinecone.DescribeIndexStats verifies what remains, and Discord gets a confirmation with the remaining vector count. Be deliberate with the input: the purge is irreversible, and Pinecone offers no undelete.

How it works

1. purge_namespace (io.kestra.plugin.pinecone.DeleteVectors) targets the namespace named by the stale_namespace input with deleteAll: true, removing every vector in it permanently.
2. readback_stats (io.kestra.plugin.pinecone.DescribeIndexStats) reads the index after the purge and returns totalVectorCount, so the confirmation reports what actually remains rather than assuming success.
3. notify posts the cleared namespace and the remaining vector count to Discord as plain scalars.
4. The errors block posts a distinct Discord alert when the purge fails, since a silently failed retention job is a compliance problem waiting to be found.
5. A disabled-by-default Schedule trigger runs the purge every Monday at 04:00.

What you get

• A one-call retention sweep for an entire namespace, no vector id bookkeeping required.
• A stats readback that turns the confirmation into evidence instead of hope.
• An explicit, human-visible input for the target namespace, so nobody purges production by editing a hardcoded string.
• An execution history that doubles as a retention log for audits.

Who it's for

• Platform teams whose Pinecone indexes accumulate experiment and demo namespaces.
• Data governance owners who need retention policies executed on a schedule, not on memory.
• ML teams isolating workloads by namespace who want the cleanup as disciplined as the ingestion.

Why orchestrate this with Kestra

Retention enforced by a script someone runs occasionally is retention that does not happen. Kestra gives the purge a schedule, an input that makes the target explicit at execution time, a readback that proves the result, and a permanent execution record. When an auditor asks when the experiments namespace was last cleared, the answer is an execution id, not a shrug.

Prerequisites

• A Pinecone index with a namespace whose contents are safe to delete permanently.
• Certainty about the namespace name: the purge removes every vector in it and cannot be undone.
• A Discord incoming webhook for confirmations and alerts.

Secrets

• PINECONE_API_KEY: Pinecone API key with delete permission on the index.
• DISCORD_WEBHOOK_URL: Discord incoming webhook URL.

Quick start

1. Add the PINECONE_API_KEY and DISCORD_WEBHOOK_URL secrets to your Kestra namespace.
2. Execute the flow against a throwaway namespace and check the Discord confirmation reports the expected remaining count.
3. Set disabled: false on the weekly trigger once the target namespace is settled.

How to extend

• Purge several stale namespaces in one run by wrapping the delete in a ForEach over a list input.
• Add a pause for human approval before the delete when the namespace name comes from an upstream system.
• Record the before and after totalVectorCount by adding a stats task ahead of the purge and including both scalars in the confirmation.
• Pair with the GDPR vector erasure blueprint when the requirement is deleting specific ids rather than whole namespaces.

Links

• Pinecone plugin documentation
• Pinecone documentation