Check for orphaned Qdrant vector points, alert Slack or auto-purge them, and log the result for RAG and data platform teams.
A RAG pipeline embeds a record once and rarely revisits it: when the source row is deleted from the primary database, nothing automatically tells Qdrant to forget its vector. The embedding lingers, fully searchable, long after the record it represents is gone, which is both stale context for retrieval and, when the source held personal data, a real compliance gap. This blueprint checks whether a batch of points flagged as orphaned by your deletion pipeline still exist in Qdrant, and only purges them once you have explicitly said this flow is allowed to.
This blueprint was created by Mohit Jeswani.
check_points (io.kestra.plugin.qdrant.Get) looks up orphaned_point_ids by ID.
Qdrant's own API simply omits a point that no longer exists from the result rather than
erroring, so an already-purged point needs no special error handling to detect.evaluate (io.kestra.plugin.core.output.OutputValues) counts and lists the flagged IDs
Qdrant still returns, then classifies the check as ALL_PURGED, VIOLATION_AUTO_PURGE, or
VIOLATION_NEEDS_HUMAN, folding confirm_purge into the classification itself.route (io.kestra.plugin.core.flow.Switch):io.kestra.plugin.qdrant.Delete (the safe remediation,
since deleting a point whose source is already gone is the already-correct outcome) and
posts a Slack notice that it happened automatically.summary logs the severity, closing the governance cycle.errors alerts Slack if the flow itself fails, for example an unreachable Qdrant host.confirm_purge to allow it.Get and Delete used
together as a compliance gate, distinct from this repo's existing Qdrant blueprints, which
focus on collection indexing and HNSW health rather than per-point lifecycle compliance.Qdrant can delete a point on request, but it has no concept of "this point's source record
was deleted elsewhere," no escalation when a deletion never arrives, and no Slack integration
of its own. Kestra keeps orphaned_point_ids and confirm_purge as flow inputs, turns a
batch existence check into a purge decision, and keeps a full execution history proving
exactly which points were checked and what happened to each one.
A local Qdrant instance can be run via the official qdrant/qdrant Docker image for a fully
local end-to-end test, no managed account required.
QDRANT_API_KEY: the Qdrant API key used by check_points and purge_points. In Kestra
OSS, provide it as the base64-encoded environment variable SECRET_QDRANT_API_KEY; leave it
unset for a local instance with no authentication configured.SLACK_WEBHOOK_URL: used by both Slack alert branches and the errors block. In Kestra
OSS, provide it as the base64-encoded environment variable SECRET_SLACK_WEBHOOK_URL.| Input | Default | Purpose |
|---|---|---|
qdrant_collection |
docs |
Collection to check for orphaned points |
orphaned_point_ids |
[] |
Point IDs whose source record has already been deleted |
confirm_purge |
false |
Authorizes this flow to delete still-present points itself |
{{ outputs.check_points.rows }}: the flagged points that still exist, as Qdrant returns
them.{{ outputs.evaluate.values.still_present_count }} / .still_present_ids: how many flagged
points are still present and their IDs, semicolon-joined.{{ outputs.evaluate.values.severity }}: ALL_PURGED, VIOLATION_AUTO_PURGE, or
VIOLATION_NEEDS_HUMAN.Get silently drops a missing ID from rows instead of erroring; this blueprint relies on
that behavior to detect an already-purged point, verified directly against the task's own
source rather than assumed from its Javadoc, which does not call this out explicitly.confirm_purge applies to the whole batch, not per point; a scheduled run with
confirm_purge: true auto-deletes every still-present point in orphaned_point_ids, so
scope one execution to one reviewed batch rather than an entire collection sweep.Delete by ids has no dry-run mode; test against a disposable collection first, since a
point deleted here cannot be recovered unless you kept the original vector and payload
elsewhere.QDRANT_API_KEY (if your instance requires one) and SLACK_WEBHOOK_URL secrets.qdrant/qdrant container, create a collection, and upsert a test point.orphaned_point_ids to that point's ID and run with confirm_purge: false; confirm
alert_violation fires.confirm_purge: true; confirm purge_points deletes it and
alert_auto_purged fires.log_all_purged fires now that the point is gone.daily trigger once orphaned_point_ids is fed from a real deletion pipeline.orphaned_point_ids automatically from the same job that deletes the source record,
instead of maintaining the list by hand.check_points and purge_points to Qdrant's filter property instead of ids, to
sweep every point whose payload still references a deleted source ID rather than a
pre-collected list.purge_points runs, to keep
an audit record of exactly what was removed and why.Drop a recording into S3 to transcribe it locally with Whisper, draft show notes and chapters with an LLM, approve them, and publish the audio and a validated RSS feed.
Join features to labels as they were known at each label's cutoff with a DuckDB ASOF join, audit the set for future rows and correlation lift, then publish it or block it.
Fetch the requested vector ids, delete them, fetch again to prove absence, and post a before-and-after audit message to Slack.
