Query icon
OutputValues icon
Labels icon
Log icon
Pause icon
Switch icon
Publish icon
RealtimeTrigger icon

Human triage for a RabbitMQ parking lot, with replay and a Postgres ledger

Triage a RabbitMQ parking lot or DLQ with Kestra. Each message is logged to Postgres and waits for a person to replay it, fixed if needed, or discard it.

Categories
Infrastructure

Parking lots and dead-letter queues fill up quietly. When someone finally looks, the usual tools move everything back at once, poison messages included, and nothing records what was replayed or why. This flow turns each parked message into a Kestra execution that records it in a Postgres ledger and pauses. Whoever is on call opens it, reads why it was parked, and resumes it with REPLAY (optionally with a corrected payload) or DISCARD and a note. The message goes back to the work queue or is dropped, and the ledger keeps the decision.

It works with any parking lot or DLQ, and is built to follow rabbitmq-retry-ladder-consumer, which parks messages with the reason and the number of attempts.

This blueprint was created by kruxarth.

How it works

  1. on_parked_message (io.kestra.plugin.amqp.RealtimeTrigger) starts one execution per message on orders.parking-lot. Executions run side by side, so a message waiting for a decision never holds up the next one.
  2. ensure_ledger creates the parked_messages table on first use.
  3. read_parked_message reads the envelope written by the consumer flow (AMQP type kestra.parked-message). Any other message, such as one RabbitMQ dead-lettered itself, is wrapped the same way, with the source queue and reason taken from its x-first-death-* headers.
  4. record_parked (io.kestra.plugin.jdbc.postgresql.Query) inserts a PENDING row with bound parameters, and show_parked_message logs the message id, the reason and the payload.
  5. await_decision (Pause) waits with three onResume inputs: decision (REPLAY or DISCARD), patched_payload and note.
  6. apply_decision (Switch):
    • REPLAY: replay (io.kestra.plugin.amqp.Publish) sends the corrected payload, or the original one, back to orders.work as a persistent message with the same message_id, and the row becomes REPLAYED.
    • DISCARD: the row becomes DISCARDED.
  7. Labels show messageId and decision (pending, replay or discard) on every execution.

What you get

  • Nothing goes back to the work queue without a decision, and each decision is kept with its note, the corrected payload and the time it was made.
  • Fix-and-replay: correct a bad field in the resume form instead of writing a script.
  • The open work at a glance: paused executions labelled decision: pending, and PENDING rows you can query or chart.
  • It keeps up with bursts: in tests, 12 messages parked at once had their rows and paused executions within 5.2 s, and 10 decisions sent through the API were applied in 1.9 s.

Who it's for

  • On-call and SRE teams who own a RabbitMQ parking lot or DLQ.
  • Backend and support teams who fix bad data and need to send the message through again.

Why orchestrate this with Kestra

  • Pause with typed onResume inputs gives you an approval form in the UI and an API endpoint, with no extra app to build.
  • Every parked message is its own execution with logs and labels, so the audit trail exists before anyone writes a report.
  • Retries on the replay publish, secrets and the Postgres plugin's bound parameters are configuration, not code.

Prerequisites

  • RabbitMQ with a parking-lot queue. Running the consumer flow once creates orders.parking-lot; otherwise set parking_lot_queue to your DLQ.

  • Postgres, reachable from Kestra:

    docker run -d --name ledger -p 5432:5432 -e POSTGRES_USER=kestra -e POSTGRES_PASSWORD=change-me -e POSTGRES_DB=messaging postgres:16-alpine
    

The variables point at localhost. If Kestra runs in Docker, use host.docker.internal instead.

Secrets

  • RABBITMQ_USERNAME and RABBITMQ_PASSWORD: a RabbitMQ user that can read the parking lot and write to replay_queue.
  • LEDGER_DB_USERNAME and LEDGER_DB_PASSWORD: a Postgres user that can create the parked_messages table and insert and update its rows.

Quick start

  1. Start Postgres, add the four secrets, and run the consumer flow once (or point parking_lot_queue at an existing queue).
  2. Save this flow. In the RabbitMQ console, publish {"order_id": "7"} to orders.parking-lot with the property message_id = order-7.
  3. A paused execution appears. Its logs show the message; click Resume, choose REPLAY, enter {"order_id": "7", "amount": 10} as the corrected payload, and add a note.
  4. Check the ledger: SELECT message_id, status, replayed_payload, decision_note FROM parked_messages;

Things to know

  • The trigger acknowledges each message once its execution is created, so orders.parking-lot looks empty while decisions are pending. The paused executions and the PENDING rows hold the messages; don't delete a paused execution before deciding.
  • The pause has no time limit. Set pauseDuration and behavior on await_decision to discard or escalate automatically.
  • A replay goes straight to replay_queue through the default exchange, and the consumer flow starts a fresh retry ladder for it.
  • To decide in bulk, call POST /api/v1/<tenant>/executions/<id>/actions/resume with the form fields decision, patched_payload and note.
  • Original AMQP headers are not carried over to the replayed message.

How to extend

  • Notify the on-call channel from onPause on await_decision, with a link to the execution.
  • Schedule a daily report of PENDING rows grouped by reason.
  • Add a REPLAY_LATER choice that republishes to one of the consumer's delay queues instead.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.