If icon
Fail icon
Subscribe icon
IonToJson icon
Script icon
Docker icon
Log icon
SlackIncomingWebhook icon
Pause icon
Loop icon
Publish icon
Schedule icon

Purge a stale MQTT retained message that re-breaks every reconnecting client

Detect stale or invalid MQTT retained messages, clear them with a zero byte retained publish behind an approval, then confirm with a new subscriber.

Categories
Infrastructure

A retained message is the broker's stored last known value for a topic, handed to every client the moment it subscribes. That is what you want on a config or command topic. It is also what hurts when the stored value is wrong. A bad config push, a firmware rollback, a decommissioned topic still handing out an old command: the value sits on the broker and re-breaks every device that reconnects, forever, with no error anywhere. The broker is doing exactly what it was told.

MQTT has one way to delete a retained message. Publish an empty payload to the same topic with the retain flag set. The broker then drops its stored value. This blueprint finds the retained messages worth deleting, then makes that publish behind a dry run and an approval, then proves the broker stopped replaying them.

Findings

Measured on Mosquitto 2.0.22 while building this blueprint.

  • io.kestra.plugin.mqtt.Publish with serdeType: STRING, from: "" and retain: true is the combination that clears a retained message. serdeType: BYTES with the same empty value fails at deserialization instead of publishing, so it cannot be used for the clear.
  • Every record io.kestra.plugin.mqtt.Subscribe writes carries the MQTT retain flag, so the audit can separate the broker's stored state from traffic that happened to arrive during the read window. A message published without the retain flag before the run is not stored at all and a later subscriber never sees it.
  • MQTT puts no timestamp on a retained message. The protocol gives you the payload and nothing else, so age has to come from the payload itself. This flow reads an ISO 8601 UTC field named by timestamp_field and says so in the log, rather than implying the broker reported an age.
  • A retained publish that a broker ACL denies does not come back as an error. The publish task succeeds, the stored value stays. Nothing in the flow notices. Re-reading the broker with a fresh client is the only check that catches it, which is why that step can fail the run.

Safety

  • Clearing a retained message is irreversible. The stored value is gone. A client that subscribes afterwards gets nothing on that topic until a fresh publish arrives. For a config topic that can mean a device sitting unconfigured, so the purge sits behind dry_run, auto_remediate plus a Pause. The Slack message says plainly what the devices lose.
  • topic_filter bounds the entire run. A topic outside it is never read and never cleared, plus a bare # or + filter is refused before the first connection.
  • max_clear_per_run caps how many topics one run may purge. The rest are reported and wait.
  • The flow never publishes a value. The only payload it ever sends is empty.

How it works

  1. guard_topic_scope refuses a broker wide filter, so the audit always has a bounded scope.
  2. read_retained (io.kestra.plugin.mqtt.Subscribe) connects with a client id built from the execution id, which the broker has never seen. On subscribe the broker immediately replays its retained message for every matching topic, with the retain flag set on each one.
  3. retained_to_json converts the collected batch to newline delimited JSON.
  4. audit_retained keeps only records whose retain flag is true, so live traffic from the read window is dropped. For each retained payload it parses the JSON, checks every field in required_payload_fields is present and compares timestamp_field against max_age_days. A payload that is not a JSON object at all is flagged too, which is the common shape of a poison retained message.
  5. report_clean logs the all clear when nothing is flagged. alert_findings posts the flagged topics to Slack either way, because the broker is handing them out right now.
  6. report_pending is the read only outcome: it names the exact publishes the flow would make.
  7. remediate_gate runs only when there are findings, auto_remediate is true and dry_run is false. It pauses for approval. An unanswered resume leaves approved null, which the gate reads as a refusal.
  8. remediate_clear_retained loops the flagged topics with concurrencyLimit: 1 and publishes a zero byte payload with retain: true to each.
  9. verify_retained connects with a second unused client id and reads the broker again. verify_audit lists which targeted topics are still replayed. Then fail_if_still_retained fails the run if any of them is. Only after that gate does notify_cleared report success.

What you get

  • A weekly read only audit of what your broker hands to every reconnecting client.
  • A purge path that is explicit about being irreversible and cannot run by accident.
  • A re-read from the broker as the proof, so a clear that silently did not land fails the run.
  • Precision: a fresh valid retained message on a neighbouring topic is never touched.
  • A Slack trail of what was flagged, what was purged and what devices lose as a result.

Who it's for

  • IoT and fleet platform teams whose devices read config from retained MQTT topics.
  • Anyone who has pushed a bad retained config and watched it re-break devices on reconnect.
  • Platform engineers decommissioning topics and finding the broker still serving them.
  • On call engineers debugging a device that misbehaves immediately after every restart.

Why orchestrate this with Kestra

Clearing retained state by hand is a one line mosquitto_pub -r -n, which is exactly the problem: it is fast, irreversible and leaves no record. Kestra turns it into a reviewed path. The audit runs on a schedule in read only mode, the purge needs two inputs plus a human approval, the re-read is a task that can fail the run. The whole sequence is one execution with logs and Slack messages attached. The same flow handles detection, the decision and the proof, which a broker CLI cannot.

Tested end to end

Run against Mosquitto 2.0.22 on Kestra 2.0.5, with mosquitto_sub confirming broker state before and after every run rather than trusting the Kestra log.

  • Seeded five topics under config/+: a stale retained config, a fresh valid retained config, a retained payload missing both required fields, a retained payload that is not JSON at all, plus one message published without the retain flag.
  • The detector saw 4 retained messages and 0 of the non retained one, then flagged exactly 3. The fresh valid one was not flagged.
  • With only the fresh valid message left, the same detector reported 0 findings, so it is not simply reporting everything it reads.
  • A non retained message published while the read window was open was collected by the subscription then dropped by the retain flag check, which is the live traffic case.
  • dry_run: true named the targets and changed nothing. mosquitto_sub showed every retained message still there afterwards.
  • max_clear_per_run: 1 against 3 findings reported 1 eligible plus 2 deferred.
  • With auto_remediate: true, dry_run: false and the pause approved, all 3 targets were cleared and the re-read found 1 retained message left, the fresh one. mosquitto_sub agreed.
  • The strongest check: a brand new subscriber received the stale value on connect before the run then received nothing on that topic after it, which is the device symptom and its cure.
  • Declining the pause published nothing. An unanswered resume behaved as a refusal.
  • A broker ACL that denies the publishing client made the clear a silent no op. The publish task still succeeded, so fail_if_still_retained failing the run is the only thing that caught it.
  • A bare # filter was refused before the first connection.
  • A broker made unreachable sent the run through the errors block to Slack.

Prerequisites

  • An MQTT broker reachable from Kestra, speaking MQTT 3.1.1 or 5.
  • A broker account allowed to subscribe to topic_filter and publish to the topics under it.
  • Payloads on the audited topics that are JSON objects carrying a timestamp field.

Secrets

  • MQTT_SERVER: broker URI, for example tcp://broker.internal:1883 or ssl://broker.internal:8883.
  • SLACK_WEBHOOK_URL: incoming webhook for the findings, purge and failure messages.

Inputs

  • mqtt_server: broker URI. Defaults to the MQTT_SERVER secret.
  • topic_filter: subscription filter that scopes the audit. Default config/+.
  • timestamp_field: payload field holding an ISO 8601 UTC timestamp. Default updated_at.
  • required_payload_fields: fields every retained payload must carry.
  • max_age_days: age at which a retained payload counts as stale. Default 30.
  • listen_duration: how long each read waits. Default PT10S.
  • max_messages: ceiling on messages collected per read. Default 200.
  • max_clear_per_run: ceiling on topics purged in one run. Default 5.
  • dry_run: report only. Default true.
  • auto_remediate: allow the purge. Default false.

Quick start

  1. Add the MQTT_SERVER and SLACK_WEBHOOK_URL secrets.
  2. Run it as is. Defaults are read only, so the first run only tells you what your broker is handing out on config/+.
  3. Read the audit log line. It gives the retained count, the flagged count, the stale cutoff and the exact topics.
  4. Confirm independently with mosquitto_sub -h <broker> -t 'config/+' -v -W 3, which prints the same retained messages the flow read.
  5. To purge, run with auto_remediate: true and dry_run: false, read the pause, then resume with approved: true.
  6. Re-run mosquitto_sub. The cleared topics print nothing and the healthy ones still print.

How to extend

  • Point topic_filter at command/+ or state/+ to audit other retained namespaces, one filter per flow so each keeps its own ceiling and schedule.
  • For payloads with no timestamp convention, keep a first seen snapshot of each topic's value in Kestra KV and derive age from when the value last changed. That measures how long the broker has served this value, not when the device produced it, so state which one a dashboard shows.
  • Republish a known good config after a clear, so devices are not left unconfigured.
  • Add a validity rule for a minimum schema_version to catch payloads that are fresh but written by a retired producer.
  • Enable the schedule once the read only output looks right. Keep it in safe mode.

Links

This blueprint was created by zkasuran.

See How

New to Kestra?

Use blueprints to kickstart your first workflows.