If icon
CreateQueue icon
Loop icon
Exit icon
OutputValues icon
Labels icon
Log icon
Request icon
Switch icon
Set icon
Delete icon
Publish icon
parked-message icon
RealtimeTrigger icon

RabbitMQ consumer with a retry ladder, deduplication and a parking lot

Consume RabbitMQ in real time with Kestra. Retries wait in TTL delay queues, duplicates are dropped, and failing messages go to a parking lot.

Categories
Data

Retrying inside a consumer holds up the queue, a nack with requeue retries in a hot loop, and a plain dead-letter queue gives up after one failure without saying why. This flow consumes a RabbitMQ queue in real time and delivers each message to an HTTP handler (an order service, an ERP endpoint, a partner webhook). A failed delivery waits in a RabbitMQ delay queue and comes back by itself after 10 seconds, then 1 minute, then 10 minutes. A message the handler rejects, or one that still fails after the last delay, goes to a parking-lot queue with the reason.

Pair it with rabbitmq-parking-lot-triage, which records every parked message in Postgres and pauses until someone decides to replay it (optionally with a corrected payload) or discard it.

This blueprint was created by kruxarth.

How it works

  1. on_message (io.kestra.plugin.amqp.RealtimeTrigger) starts one execution per message on orders.work. The payload is read as a string, so a message that is not JSON cannot stop the trigger.
  2. A manual run has no trigger, so declare_topology (If) creates the queues and stops: the work queue, the parking lot, and one orders.work.retry-<seconds>s queue per entry in retry_delays_seconds (Loop over CreateQueue). Each delay queue has x-message-ttl and dead-letters through the default exchange straight into the work queue, so a retry never reaches other queues bound to the producer's exchange.
  3. identify takes the AMQP message_id, or a hash of the body when there is none, and drop_duplicate ends the execution if that message was delivered within dedupe_window (a KV key with a TTL).
  4. count_attempt reads the attempt number from KV. RabbitMQ's x-death header would hold it, but the AMQP plugin returns every header as a "key=value" string.
  5. deliver (io.kestra.plugin.core.http.Request) POSTs the payload with Idempotency-Key and X-Delivery-Attempt headers, a 5-second connect timeout and a 30-second read timeout.
  6. decide and route (Switch) act on the result:
    • 2xx: delivered. The message id is remembered for deduplication.
    • 408, 425, 429, 5xx or no response, with delays left: schedule_retry (io.kestra.plugin.amqp.Publish) puts a persistent copy on the next delay queue. The execution ends in WARNING.
    • Any other 4xx, or the last delay used up: park publishes the message to orders.parking-lot with the reason, the number of attempts and the last status code.
  7. Every execution carries messageId, attempt and outcome labels.

What you get

  • Backoff that costs nothing while it waits: the delays live in RabbitMQ, not in sleeping executions. In tests, retries arrived within 1 s of the configured 10 s, 60 s and 600 s.
  • Nothing lost when RabbitMQ restarts: retries are persistent messages in durable queues, the trigger reconnects by itself, and a publish that hits an outage is retried with backoff for about 15 minutes.
  • Duplicates dropped, whether the copy arrives seconds later or at the same instant, without calling the handler.
  • The full history of a message from one label filter: one execution per attempt, with its status code and error.
  • Poison messages leave the work queue at once and never block the ones behind them.

Who it's for

  • Backend and platform teams that push queue messages to HTTP services: partner webhooks, order and ERP sync, payment callbacks.
  • Anyone who needs an audit trail of every delivery attempt.

Why orchestrate this with Kestra

  • The retry and parking policy is a few YAML variables instead of code repeated in every consumer.
  • Each attempt is an execution with its own logs, state and labels, not a line lost in a container log.
  • Real-time triggers, KV with TTL, concurrency control and task retries are built in, and the triage flow adds a human decision without another service.

Prerequisites

  • RabbitMQ 3.12 or later, reachable from Kestra:

    docker run -d --name rabbitmq -p 5672:5672 -p 15672:15672 -e RABBITMQ_DEFAULT_USER=kestra -e RABBITMQ_DEFAULT_PASS=change-me rabbitmq:4-management-alpine
    
  • An HTTP endpoint that answers 2xx once the message is processed, 4xx when it never will be, and 5xx or 429 to be tried later, and that uses Idempotency-Key to ignore a second copy. To try the flow without one, run docker run -d --name handler -p 8000:8080 mccutchen/go-httpbin and set handler_url to http://localhost:8000/anything/orders.

The variables point at localhost. If Kestra runs in Docker, use host.docker.internal instead (on Linux, add extra_hosts: ["host.docker.internal:host-gateway"] to the Kestra service).

Secrets

  • RABBITMQ_USERNAME and RABBITMQ_PASSWORD: a RabbitMQ user allowed to declare queues and to read and write orders.* in the virtual host.

Quick start

  1. Start RabbitMQ and a handler, and add the two secrets.
  2. Execute the flow once to create the queues. Until then, the trigger logs that orders.work does not exist.
  3. In the RabbitMQ console (http://localhost:15672), open Queues > orders.work > Publish message, add the property message_id = order-1 and the payload {"order_id": "1", "amount": 42}. An execution labelled outcome: delivered appears within a second.
  4. Publish the same message again: it ends as outcome: duplicate.
  5. Stop the handler and publish order-2. Watch orders.work.retry-10s and then retry-60s hold it, start the handler again, and the next attempt delivers it.

Things to know

  • The trigger acknowledges a message once its execution is created, so the execution is then its only copy. If a publish still fails after its retries, the errors handler logs it; replay the execution once RabbitMQ is back.
  • concurrency (limit 1, QUEUE) handles one message at a time, about 0.4 s each in tests, which makes deduplication exact. A higher limit is faster, but two copies arriving together may then both be delivered.
  • A request that timed out may still have been processed by the handler. The retry carries the same Idempotency-Key so the handler can drop it.
  • Retries and parked messages keep the payload, message_id and content type, but not the original headers, which the AMQP plugin cannot read back intact.
  • Changing a delay creates a queue with a new name instead of failing on the existing one. Delete the old queue once it is empty.

How to extend

  • Tune retry_delays_seconds: add a rung, or remove one for faster parking.
  • Add 'x-queue-type': 'quorum' to the work and parking-lot queues on a replicated cluster.
  • Route on trigger.routingKey to send different message types to different handlers.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.