Request icon
Query icon
OutputValues icon
Log icon
Switch icon
Fail icon
Pause icon
If icon
Loop icon
PublishMessage icon
Sleep icon
CompleteJob icon
SlackIncomingWebhook icon
Trigger icon
Schedule icon

Backfill Camunda Messages Lost Between an Outbox and a Catch Event

Detect Camunda instances parked past an SLA with unsent outbox events, republish each message serially and verify recovery.

Categories
Infrastructure

Safety

Republishing an event can advance an order, charge a customer or trigger another external side effect. The flow therefore limits the target to one exact process definition and message name, requires both an old active message subscription and an old pending outbox row with the same correlation key, refuses a candidate set above max_candidates and defaults to read-only. Approval is a stale snapshot: the engine and outbox are read again immediately before the write. Every message uses the outbox event ID as Camunda's deduplication ID. A serial loop limits pressure on the broker. After publishing, the flow reads Camunda's subscriptions again and fails if any matching instance is still parked.

Guard What it stops
Exact process and message inputs Cross-workflow or cross-event repair
SLA on both engine and outbox rows Repairing a fresh event or a recently started instance
Matching correlation key and event ID Sending an event to the wrong order or twice
Candidate ceiling and serial loop A producer outage becoming a burst of writes
Approval plus live re-read Acting on a stale incident list
Post-write subscription read Treating a green command as recovered work

How it works

  1. read_subscriptions (io.kestra.plugin.core.http.Request) reads CREATED Camunda message subscriptions for the exact process and message.
  2. read_instances (io.kestra.plugin.core.http.Request) reads ACTIVE instances older than the SLA. read_outbox (io.kestra.plugin.jdbc.postgresql.Query) reads only old PENDING rows whose published_at is null.
  3. classify intersects the native reads on process instance key and correlation key. The plan includes the event ID and JSON variables and is logged for review.
  4. With auto_remediate: true and dry_run: false, approval pauses. reread_subscriptions, reread_instances and reread_outbox rebuild the plan after resume. stale_plan_guard fails if its stable signature changed.
  5. republish (io.kestra.plugin.core.flow.Loop, concurrencyLimit: 1) calls io.kestra.plugin.camunda.PublishMessage once per candidate with the event ID as messageId and PT0S TTL.
  6. verify_subscriptions and verify_instances read the engine again. fail_still_parked fails if any matching CREATED subscription remains. A Camunda Trigger can invoke the same flow for a repair service task; complete_job (io.kestra.plugin.camunda.CompleteJob) reports success only after verification.
  7. errors posts a Slack alert with the execution ID whenever detection, approval, writing or verification fails.

What you get

  • A dry-run incident packet showing exact process instance keys, message names, correlation keys and outbox event IDs.
  • A bounded approval path that republishes only a live, revalidated set.
  • A native Camunda message command with serial concurrency and deduplication IDs.
  • A post-write proof from Camunda's own subscription search, plus a failed execution when recovery did not land.

Who it's for

Platform teams operating Camunda 8 order, payment or fulfillment processes with a transactional outbox and a broker handoff. It is for an incident where the producer committed an event but the broker or worker path lost it while the process instance remained at a message catch event.

Why orchestrate this with Kestra

A SQL script can find an old outbox row and a Camunda command can publish a message, but neither one provides an approval snapshot, serial bounded fan-out, stale-plan refusal, task lineage and an alert when the instance stays parked. Kestra keeps the detector, the typed Camunda write and the independent recovery read in one auditable execution.

Tested end to end

Kestra 2.0.5, Camunda 8.9.19 (camunda/camunda@sha256:174f43860c96f25e259785b0d65bc245fbcb20e3d8ff39eef8ff01ae888813ed, one broker and one partition with RDBMS secondary storage), PostgreSQL 16.5 (postgres:16-alpine), plugin-camunda 1.0.3, plugin-jdbc-postgres 2.1.0 and plugin-slack 2.1.0. The fixture deployed a message-catch process and a separate protected process, inserted one old pending outbox row and dropped its broker delivery. Every result below was read from Camunda REST v2 and PostgreSQL with independent probes.

Case Execution Result
Healthy control 19XPXqqe4GRTcH6Tv33iWY 0 candidates; instance stayed ACTIVE with a CREATED subscription
Degraded dry run 1lk4L7hhs3AXQazPnc8F14 1 parked instance and 1 matching old outbox row shown; 0 publishes
Declined approval 3l26kE9uxzDQoabzvp2Lqg 1 candidate; approval false; 0 publishes; instance stayed parked
Remediation 4piUFoc9J09PGIa6n8dFmV 1 PublishMessage; target subscription became CORRELATED; instance COMPLETED
Repeat 7iCpvPPLp4DHjjGG8rqLRZ 0 candidates after recovery; 0 duplicate messages
Ineffective write 3z99G68xoO8pmjXXlz8XUZ Malformed variables caused FAILED; instance stayed ACTIVE with CREATED subscription
Race or partial 3GeYanS6t96t1FX94mMazr Outbox changed during approval; stale signature failed before publish

The exact execution IDs, request payloads, engine snapshots and measured counts are in the evidence packet for this blueprint.

Prerequisites

  • Camunda 8.9.19 with REST v2 process and message-subscription search, publish-message permission and a reachable Docker network.
  • PostgreSQL 16 with an outbox_events table containing event_id, message_name, correlation_key, payload, status, published_at and created_at.
  • Kestra 2.0.5 with Camunda, PostgreSQL JDBC, Slack and Docker runner plugins.
  • Minimum permissions: read process instances and message subscriptions, publish messages and activate or complete the optional repair job; PostgreSQL SELECT on outbox_events.

Secrets

  • SLACK_WEBHOOK_URL: required by the error alert.
  • Use secret-backed Camunda and PostgreSQL credentials in production.

Inputs

Input Default Purpose
camunda_rest_address http://localhost:8080 Camunda REST base
outbox_jdbc_url jdbc:postgresql://localhost:5432/outbox Producer outbox
process_definition_id order_fulfillment Exact process allowlist
message_name order-payment-authorized Exact message allowlist
sla_minutes 15 Age floor for both records
max_candidates 10 Refusal ceiling
auto_remediate false Explicit write permission
dry_run true Explicit safe mode

Quick start

  1. Start Camunda and PostgreSQL on a reachable Docker network and create the outbox_events table using the schema required above.
  2. Add SLACK_WEBHOOK_URL and production connection secrets in Kestra.
  3. Import this flow into namespace company.team and run it with the defaults for a dry run.
  4. Review the candidate keys and event IDs in log_plan. Set auto_remediate: true and dry_run: false only for the approved incident, then answer the Pause with approved: true.
  5. Require a SUCCESS execution and an independent Camunda read showing zero matching CREATED subscriptions before closing the incident.

How to extend

Add another process and message only as a separately reviewed allowlist entry. If an outbox schema stores variables in columns rather than JSON, change the single projection in read_outbox and preserve the stable event ID. For a large backlog, shard by process or incident window and keep each run below its ceiling. Keep the trigger disabled until its job type is deployed with a dedicated worker contract.

Limits: the flow cannot prove that the original producer did not already publish an event outside the outbox's published_at marker and it does not mark the outbox row as published. Operators must reconcile duplicate business effects with the producer's idempotency policy. Camunda secondary search is eventually consistent, so the post-write read waits three seconds and a failed verification remains an incident.

Links

This blueprint was created by zkasuran.

See How

New to Kestra?

Use blueprints to kickstart your first workflows.