New to Kestra?
Use blueprints to kickstart your first workflows.
Detect MongoDB documents referencing a deleted parent with $lookup, delete the aged orphans behind an approval, then re-join to prove none survived.
MongoDB has no foreign keys. A manual reference is just a field holding another document's
key, so deleting a parent is allowed while children still point at it. The moment a customer
record goes, every order carrying that customerId references a key nothing resolves. Nothing
errors. The orders still read back fine on their own. The damage shows up in the joins: a
$lookup on those orders returns an empty array, an aggregation that unwinds the join drops
the row entirely, a revenue report grouped by customer quietly loses the money. Deletes land
through an admin console, a GDPR erasure job, a retention sweep or a support script, so the
children outlive their parent in almost every application that models references by hand.
This blueprint is the repair for that, inside MongoDB itself. It joins the child collection to
the parent collection server side with $lookup, treats an empty join array as a dangling
reference, then splits the dangling set by age. A child written in the last few minutes is
reported but never deleted, because a child can legitimately be written just before its parent
or land in a multi document insert that is still in flight. The aged orphans are deleted by
_id behind a safe mode gate and an approval. Then the same join runs again. If an orphan old
enough to judge is still there, the run fails.
This blueprint was created by zkasuran.
auto_remediate true, dry_run false, then a human answering the Pause. Three conditions,
all of them, before a document is removed.min_orphan_age_minutes is counted, listed, never deleted. Without it a sweep running
during a multi document insert deletes rows whose parent was about to arrive.$lookup compares BSON types strictly. A string customerId against an ObjectId _id
matches nothing, which makes every referencing child look orphaned. type_probe groups both
sides by BSON type first and the write path refuses outright when the two type sets are
disjoint.max_orphan_ratio is the second rail on the same risk. When more than half the referencing
children look orphaned, the likelier fault is the join, not the data, so the run fails
instead of deleting._id list, never on the reference field. A child whose
parent exists cannot be caught by a stale filter. Neither can a fresh child the age gate
spared.delete_limit per run and batched by batch_size. If more are aged
than the cap, the re-join fails until a later run finishes, so a partial clear is never
reported as done.DELETE_MANY, so on orders that is revenue history with no backup behind it. That is why the
type probe, the share ceiling, the age gate, auto_remediate, dry_run and the Pause all
sit in front of it.check_inputs (io.kestra.plugin.core.flow.If) fails the run before any read when the two
collections are the same, a limit is below 1, the age gate is negative or the orphan share
ceiling sits outside 0 to 1.type_probe (io.kestra.plugin.mongodb.Aggregate) groups the reference field by BSON type
on the child side, then $unionWith adds the same grouping of the parent key field. This is
the join compatibility check.log_types (io.kestra.plugin.core.log.Log) prints both type sets with the disjoint
verdict, because task outputs are not in the execution JSON.detect (io.kestra.plugin.mongodb.Aggregate) runs the $lookup from the child collection
into the parent collection, marks a child with an empty join array as dangling, derives the
child write time from age_field or from the timestamp inside its ObjectId, then compares
it with a cutoff frozen on execution.startDate. One $facet returns the referencing
total, the orphan count, the aged count, the spared count, the capped id list the delete
acts on, the orphan share and a sample of the oldest orphans.report (io.kestra.plugin.core.log.Log) logs those numbers with the cutoff, so a spared
child can be checked by hand.route (io.kestra.plugin.core.flow.If): zero orphans logs clean and stops. One or more
sends alert to Slack then prints planned, which states what a remediation run would do
on every run including a dry one.remediate_gate (io.kestra.plugin.core.flow.If) opens only with auto_remediate true and
dry_run false. Inside it safety_check fails the run through
io.kestra.plugin.core.execution.Fail on disjoint join types or an orphan share above the
ceiling. Otherwise hold (io.kestra.plugin.core.flow.Pause) waits for an answer.remediate (io.kestra.plugin.core.flow.Loop, concurrencyLimit: 1) walks the
id batches and purge (io.kestra.plugin.mongodb.Delete, DELETE_MANY) removes each batch
by _id. With nothing aged the run logs nothing_to_delete.reverify (io.kestra.plugin.mongodb.Aggregate) runs the join again against the same
frozen cutoff and verify_gate fails the run if any aged orphan survived. Without approval
declined records the decision and nothing changes.errors posts to Slack when any task fails, for example when MongoDB is unreachable._id, capped per run.$lookup aggregations lose rows, where the join looks correct because
the missing parent is invisible from the child side.Referential integrity in MongoDB is the application's job, which means nobody owns it once the
application that wrote the reference has moved on. The usual answer is a script somebody runs
after an incident. This flow makes it scheduled, reviewable work. The detection is one server
side pipeline, so a large collection never leaves the database. The age gate turns a race into
a documented decision. The delete sits behind auto_remediate, dry_run, a type probe, a
share ceiling then a Pause with the answer recorded on the execution. The batches are
serialised by concurrencyLimit: 1. The re-join turns the delete into a checked outcome rather
than a hopeful one. The Slack alert and the error path are the same ones the rest of your flows
use.
On Kestra 2.0.5 OSS against a MongoDB 7.0 container, on the final file. appdata.customers
held 3 parents, appdata.orders held 8 children whose customerId all resolved. Deleting one
customer orphaned 3 of those orders. A fourth orphan was then written fresh, with no parent at
all, to test the age gate. Every before and after number was read back with mongosh, not only
from the Kestra log.
| Run | Inputs | Result |
|---|---|---|
| Clean | defaults on intact data | 8 referencing, 0 dangling. clean logged, no alert, SUCCESS |
| Detect | defaults after deleting one customer | 3 dangling, 3 aged, 0 spared, share 0.375. Slack alert, SUCCESS |
| Dry run | the same | Reports the delete it would make. mongosh still 8 orders, 3 orphans |
| Armed but dry | auto_remediate: true, dry_run: true |
Gate stayed closed, no Pause. mongosh unchanged |
| Declined | auto_remediate: true, dry_run: false, resumed approved: false |
Paused, then declined. mongosh still 8 orders, 3 orphans |
| Age gate | a fourth orphan written fresh, min_orphan_age_minutes: 15, approved |
3 aged orphans deleted, the fresh one spared. mongosh 9 orders with 4 orphans before, 6 orders with 1 orphan after |
| Nothing aged | only the fresh orphan left, approved | nothing_to_delete, no delete issued at all |
| Share ceiling | a second parent deleted, max_orphan_ratio: 0.5 |
Share 0.6667, refuse_delete failed the run before the Pause, nothing deleted |
| Re-verify fails | max_orphan_ratio: 0.9, delete_limit: 1 with 3 aged orphans, approved |
1 deleted, re-join found 2 aged, verify_gate failed the run, errors alerted. mongosh 6 orders to 5 |
| Type mismatch | a child collection whose reference is a string against an ObjectId key |
Types disjoint, refuse_delete failed the run, all 3 documents still present |
| Write time field | age_field: createdAt instead of the ObjectId timestamp |
Same 2 aged plus 1 spared split, read from createdAt to the millisecond |
| Full clear | min_orphan_age_minutes: 0, approved |
All 3 orphans deleted, re-join 0, SUCCESS. mongosh 5 orders to 2 |
| Bad input | max_orphan_ratio: 1.5 |
check_inputs failed the run before any read |
| Unreachable | connection_uri pointed at a dead host |
type_probe failed, errors posted to Slack, execution FAILED |
The clean run and the detect run isolate it. Nothing changed between them except one deleted
parent: the intact collection returned 0 dangling out of 8 referencing children, the same flow
with the same inputs then returned 3. mongosh confirmed both by running the same $lookup
plus $match on an empty join itself. The total document count is the other half of the proof:
9 orders with 4 orphans before the age gate run, 6 orders with 1 orphan after, so exactly the
3 aged orphans went while the fresh one stayed. The re-verify gate was proven the other way by
capping delete_limit at 1, which left 2 aged orphans behind then failed the run.
_id is a stock ObjectId. The delete round trips ids through
$oid. With no age_field the write time comes from the timestamp inside the ObjectId._id, so the join is served without a collection
scan. An index on the child reference field helps the sample sorts.docker run -d --name mdb --network mynet mongo:7.0, seed a parent collection
plus a child collection referencing it, delete one parent, then run the flow with
connection_uri: mongodb://mdb:27017.MONGODB_URI: the MongoDB connection string. Read when connection_uri is left empty, which
is how the flow should run in production. In Kestra OSS, provide it as the base64-encoded
environment variable SECRET_MONGODB_URI.SLACK_WEBHOOK_URL: used by alert and the errors block. Provide it as
SECRET_SLACK_WEBHOOK_URL.| Input | Default | Purpose |
|---|---|---|
connection_uri |
empty | Plain connection string for a local test. Empty reads MONGODB_URI |
database |
appdata |
The database holding both collections |
child_collection |
orders |
The collection that carries the references |
parent_collection |
customers |
The collection the references point at |
reference_field |
customerId |
The field on a child holding the parent key |
parent_key_field |
_id |
The field on a parent the reference resolves against |
age_field |
empty | Child write time field. Empty takes it from the ObjectId _id |
min_orphan_age_minutes |
15 |
A dangling child younger than this is reported, never deleted |
max_orphan_ratio |
0.5 |
Refuse to delete above this orphan share of referencing children |
scan_limit |
25 |
How many orphans the report lists. Counts stay exact |
delete_limit |
50000 |
Ceiling on aged orphans deleted per run |
batch_size |
500 |
Orphan ids per delete batch |
approval_timeout |
PT2H |
How long the pause waits for an answer |
auto_remediate |
false |
Allow the delete |
dry_run |
true |
Must be false for a delete to apply |
MONGODB_URI and SLACK_WEBHOOK_URL secrets.database, the two collection names, reference_field then parent_key_field. Run
the flow. It reports the dangling references, how many are old enough to judge, the orphan
share. It deletes nothing.log_types first. If the two BSON type sets are disjoint, fix the reference types.
There is nothing to clean up, the join itself is wrong.auto_remediate: true with dry_run: false, run it again, then answer the pause. The
aged orphans are deleted and the re-join proves none is left.daily schedule, which always runs in safe mode.$merge of the targeted orphans into a quarantine collection
ahead of the delete, so a wrong parent deletion is recoverable.purge for io.kestra.plugin.mongodb.Update with
UPDATE_MANY to point the orphans at a placeholder parent, where the child is worth keeping.io.kestra.plugin.core.flow.Loop
over a list of child, parent, field triples, with concurrencyLimit: 1 on the part that
deletes.Pause with a Slack approval. Or drop it for a collection you are sure about.$lookup: https://www.mongodb.com/docs/manual/reference/operator/aggregation/lookup/