If icon
Fail icon
Aggregate icon
Log icon
SlackIncomingWebhook icon
Pause icon
Loop icon
Delete icon
Schedule icon

MongoDB Dangling Reference Orphan Cleanup Guard

Detect MongoDB documents referencing a deleted parent with $lookup, delete the aged orphans behind an approval, then re-join to prove none survived.

Categories
DataInfrastructure

MongoDB has no foreign keys. A manual reference is just a field holding another document's key, so deleting a parent is allowed while children still point at it. The moment a customer record goes, every order carrying that customerId references a key nothing resolves. Nothing errors. The orders still read back fine on their own. The damage shows up in the joins: a $lookup on those orders returns an empty array, an aggregation that unwinds the join drops the row entirely, a revenue report grouped by customer quietly loses the money. Deletes land through an admin console, a GDPR erasure job, a retention sweep or a support script, so the children outlive their parent in almost every application that models references by hand.

This blueprint is the repair for that, inside MongoDB itself. It joins the child collection to the parent collection server side with $lookup, treats an empty join array as a dangling reference, then splits the dangling set by age. A child written in the last few minutes is reported but never deleted, because a child can legitimately be written just before its parent or land in a multi document insert that is still in flight. The aged orphans are deleted by _id behind a safe mode gate and an approval. Then the same join runs again. If an orphan old enough to judge is still there, the run fails.

This blueprint was created by zkasuran.

Safety

  • Read-only by default. The first run with the defaults only counts and reports. A delete needs auto_remediate true, dry_run false, then a human answering the Pause. Three conditions, all of them, before a document is removed.
  • The age gate is the part that stops a race from becoming data loss. A dangling child younger than min_orphan_age_minutes is counted, listed, never deleted. Without it a sweep running during a multi document insert deletes rows whose parent was about to arrive.
  • A $lookup compares BSON types strictly. A string customerId against an ObjectId _id matches nothing, which makes every referencing child look orphaned. type_probe groups both sides by BSON type first and the write path refuses outright when the two type sets are disjoint.
  • max_orphan_ratio is the second rail on the same risk. When more than half the referencing children look orphaned, the likelier fault is the join, not the data, so the run fails instead of deleting.
  • The delete filters on the targeted _id list, never on the reference field. A child whose parent exists cannot be caught by a stale filter. Neither can a fresh child the age gate spared.
  • A child missing the reference field or holding null in it, is never read as an orphan. Only documents that actually claim a parent are judged.
  • The delete is capped at delete_limit per run and batched by batch_size. If more are aged than the cap, the re-join fails until a later run finishes, so a partial clear is never reported as done.
  • What a wrong delete costs: the child documents are gone. There is no undo on a DELETE_MANY, so on orders that is revenue history with no backup behind it. That is why the type probe, the share ceiling, the age gate, auto_remediate, dry_run and the Pause all sit in front of it.

How it works

  1. check_inputs (io.kestra.plugin.core.flow.If) fails the run before any read when the two collections are the same, a limit is below 1, the age gate is negative or the orphan share ceiling sits outside 0 to 1.
  2. type_probe (io.kestra.plugin.mongodb.Aggregate) groups the reference field by BSON type on the child side, then $unionWith adds the same grouping of the parent key field. This is the join compatibility check.
  3. log_types (io.kestra.plugin.core.log.Log) prints both type sets with the disjoint verdict, because task outputs are not in the execution JSON.
  4. detect (io.kestra.plugin.mongodb.Aggregate) runs the $lookup from the child collection into the parent collection, marks a child with an empty join array as dangling, derives the child write time from age_field or from the timestamp inside its ObjectId, then compares it with a cutoff frozen on execution.startDate. One $facet returns the referencing total, the orphan count, the aged count, the spared count, the capped id list the delete acts on, the orphan share and a sample of the oldest orphans.
  5. report (io.kestra.plugin.core.log.Log) logs those numbers with the cutoff, so a spared child can be checked by hand.
  6. route (io.kestra.plugin.core.flow.If): zero orphans logs clean and stops. One or more sends alert to Slack then prints planned, which states what a remediation run would do on every run including a dry one.
  7. remediate_gate (io.kestra.plugin.core.flow.If) opens only with auto_remediate true and dry_run false. Inside it safety_check fails the run through io.kestra.plugin.core.execution.Fail on disjoint join types or an orphan share above the ceiling. Otherwise hold (io.kestra.plugin.core.flow.Pause) waits for an answer.
  8. On approval remediate (io.kestra.plugin.core.flow.Loop, concurrencyLimit: 1) walks the id batches and purge (io.kestra.plugin.mongodb.Delete, DELETE_MANY) removes each batch by _id. With nothing aged the run logs nothing_to_delete.
  9. reverify (io.kestra.plugin.mongodb.Aggregate) runs the join again against the same frozen cutoff and verify_gate fails the run if any aged orphan survived. Without approval declined records the decision and nothing changes.
  10. errors posts to Slack when any task fails, for example when MongoDB is unreachable.

What you get

  • The exact number of child documents whose reference resolves to nothing, which no MongoDB command reports for you, because the database never knew the reference existed.
  • A Slack alert the moment a reference goes dangling, before a report joining on it starts dropping rows.
  • An age gate that separates a real orphan from a write still in flight, so a cleanup running next to a live writer does not delete a child whose parent is one insert behind.
  • A join compatibility probe that catches the mismatch that makes a naive orphan sweep delete the whole collection.
  • A gated, approved delete of the aged orphans by _id, capped per run.
  • A re-join that fails the run when an aged orphan survived, so a refused write or a concurrent writer is caught rather than hidden.

Who it's for

  • Teams modelling one to many with a manual reference, which is most MongoDB applications, with deletes arriving from a console, an erasure request or a retention job.
  • Data engineers whose $lookup aggregations lose rows, where the join looks correct because the missing parent is invisible from the child side.
  • Anyone preparing a collection for stricter integrity checks who has to find out how much already dangles.

Why orchestrate this with Kestra

Referential integrity in MongoDB is the application's job, which means nobody owns it once the application that wrote the reference has moved on. The usual answer is a script somebody runs after an incident. This flow makes it scheduled, reviewable work. The detection is one server side pipeline, so a large collection never leaves the database. The age gate turns a race into a documented decision. The delete sits behind auto_remediate, dry_run, a type probe, a share ceiling then a Pause with the answer recorded on the execution. The batches are serialised by concurrencyLimit: 1. The re-join turns the delete into a checked outcome rather than a hopeful one. The Slack alert and the error path are the same ones the rest of your flows use.

Tested end to end

On Kestra 2.0.5 OSS against a MongoDB 7.0 container, on the final file. appdata.customers held 3 parents, appdata.orders held 8 children whose customerId all resolved. Deleting one customer orphaned 3 of those orders. A fourth orphan was then written fresh, with no parent at all, to test the age gate. Every before and after number was read back with mongosh, not only from the Kestra log.

Run Inputs Result
Clean defaults on intact data 8 referencing, 0 dangling. clean logged, no alert, SUCCESS
Detect defaults after deleting one customer 3 dangling, 3 aged, 0 spared, share 0.375. Slack alert, SUCCESS
Dry run the same Reports the delete it would make. mongosh still 8 orders, 3 orphans
Armed but dry auto_remediate: true, dry_run: true Gate stayed closed, no Pause. mongosh unchanged
Declined auto_remediate: true, dry_run: false, resumed approved: false Paused, then declined. mongosh still 8 orders, 3 orphans
Age gate a fourth orphan written fresh, min_orphan_age_minutes: 15, approved 3 aged orphans deleted, the fresh one spared. mongosh 9 orders with 4 orphans before, 6 orders with 1 orphan after
Nothing aged only the fresh orphan left, approved nothing_to_delete, no delete issued at all
Share ceiling a second parent deleted, max_orphan_ratio: 0.5 Share 0.6667, refuse_delete failed the run before the Pause, nothing deleted
Re-verify fails max_orphan_ratio: 0.9, delete_limit: 1 with 3 aged orphans, approved 1 deleted, re-join found 2 aged, verify_gate failed the run, errors alerted. mongosh 6 orders to 5
Type mismatch a child collection whose reference is a string against an ObjectId key Types disjoint, refuse_delete failed the run, all 3 documents still present
Write time field age_field: createdAt instead of the ObjectId timestamp Same 2 aged plus 1 spared split, read from createdAt to the millisecond
Full clear min_orphan_age_minutes: 0, approved All 3 orphans deleted, re-join 0, SUCCESS. mongosh 5 orders to 2
Bad input max_orphan_ratio: 1.5 check_inputs failed the run before any read
Unreachable connection_uri pointed at a dead host type_probe failed, errors posted to Slack, execution FAILED

Does the detector actually fail on orphans

The clean run and the detect run isolate it. Nothing changed between them except one deleted parent: the intact collection returned 0 dangling out of 8 referencing children, the same flow with the same inputs then returned 3. mongosh confirmed both by running the same $lookup plus $match on an empty join itself. The total document count is the other half of the proof: 9 orders with 4 orphans before the age gate run, 6 orders with 1 orphan after, so exactly the 3 aged orphans went while the fresh one stayed. The re-verify gate was proven the other way by capping delete_limit at 1, which left 2 aged orphans behind then failed the run.

Prerequisites

  • A reachable MongoDB instance and a user that can read both collections then run a delete on the child collection.
  • A child collection whose _id is a stock ObjectId. The delete round trips ids through $oid. With no age_field the write time comes from the timestamp inside the ObjectId.
  • An index on the parent key field, normally _id, so the join is served without a collection scan. An index on the child reference field helps the sample sorts.
  • Local testing: docker run -d --name mdb --network mynet mongo:7.0, seed a parent collection plus a child collection referencing it, delete one parent, then run the flow with connection_uri: mongodb://mdb:27017.

Secrets

  • MONGODB_URI: the MongoDB connection string. Read when connection_uri is left empty, which is how the flow should run in production. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_MONGODB_URI.
  • SLACK_WEBHOOK_URL: used by alert and the errors block. Provide it as SECRET_SLACK_WEBHOOK_URL.

Inputs

Input Default Purpose
connection_uri empty Plain connection string for a local test. Empty reads MONGODB_URI
database appdata The database holding both collections
child_collection orders The collection that carries the references
parent_collection customers The collection the references point at
reference_field customerId The field on a child holding the parent key
parent_key_field _id The field on a parent the reference resolves against
age_field empty Child write time field. Empty takes it from the ObjectId _id
min_orphan_age_minutes 15 A dangling child younger than this is reported, never deleted
max_orphan_ratio 0.5 Refuse to delete above this orphan share of referencing children
scan_limit 25 How many orphans the report lists. Counts stay exact
delete_limit 50000 Ceiling on aged orphans deleted per run
batch_size 500 Orphan ids per delete batch
approval_timeout PT2H How long the pause waits for an answer
auto_remediate false Allow the delete
dry_run true Must be false for a delete to apply

Quick start

  1. Add the MONGODB_URI and SLACK_WEBHOOK_URL secrets.
  2. Set database, the two collection names, reference_field then parent_key_field. Run the flow. It reports the dangling references, how many are old enough to judge, the orphan share. It deletes nothing.
  3. Read log_types first. If the two BSON type sets are disjoint, fix the reference types. There is nothing to clean up, the join itself is wrong.
  4. Set auto_remediate: true with dry_run: false, run it again, then answer the pause. The aged orphans are deleted and the re-join proves none is left.
  5. Enable the daily schedule, which always runs in safe mode.

How to extend

  • Archive before deleting: add a $merge of the targeted orphans into a quarantine collection ahead of the delete, so a wrong parent deletion is recoverable.
  • Repair instead of delete: swap purge for io.kestra.plugin.mongodb.Update with UPDATE_MANY to point the orphans at a placeholder parent, where the child is worth keeping.
  • Guard several reference pairs by wrapping the tasks in a io.kestra.plugin.core.flow.Loop over a list of child, parent, field triples, with concurrencyLimit: 1 on the part that deletes.
  • Run it from the deletion side: trigger the flow after the job that deletes parents, so the children are cleaned in the same window.
  • Replace the Pause with a Slack approval. Or drop it for a collection you are sure about.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.