Log icon
If icon
Generate icon
JsonObjectGenerator icon
Webhook icon

Stale Event Processing Delay Sentinel

Detect stale or delayed events with a configurable timestamp threshold, log freshness results, and generate structured monitoring events.

Categories
Core

How it works

Event-driven workflows can process outdated events when upstream systems are delayed, queues build up, or event delivery is interrupted. This Blueprint receives events through a webhook and compares their timestamps against a configurable maximum processing delay.

  1. The webhook trigger receives an incoming event.
  2. The workflow logs the received payload.
  3. It checks whether the payload contains an event_timestamp.
  4. If the timestamp exists, it compares the event age against max_delay_minutes.
  5. It logs whether the event is fresh or stale and generates a structured monitoring event for either outcome.
  6. If the timestamp is missing, it logs a warning.

What you get

  • Detection of stale or delayed events.
  • A configurable processing-delay threshold.
  • Separate handling for fresh, stale, and missing-timestamp events.
  • Structured JSON records for fresh and stale events.
  • Execution ID, event timestamp, delay threshold, and check time in generated records.
  • Execution logs that help identify delayed event delivery.

Who it's for

  • Platform engineers operating event-driven workflows.
  • Backend engineers integrating webhooks and asynchronous event producers.
  • Reliability engineers investigating delayed event processing.
  • Teams that need to identify outdated payloads before downstream processing.

Why orchestrate this with Kestra

Kestra combines webhook reception, timestamp validation, conditional branching, logging, and structured event generation in one workflow. Execution history makes it easier to inspect how an event was classified. The flow can also be extended with notifications or downstream remediation tasks.

Prerequisites

  • A running Kestra instance.
  • Network access from the event producer to the Kestra webhook endpoint.
  • The Core plugin for webhook triggers, conditional flow control, and logs.
  • The DataGen plugin for structured JSON event generation.

Secrets

The default workflow does not require API credentials or other secrets. Protect the webhook endpoint according to your Kestra deployment's security requirements.

Quick start

  1. Import the flow into your Kestra namespace.
  2. Configure max_delay_minutes; it defaults to 15 minutes.
  3. Send a JSON payload containing an ISO-8601 event_timestamp to the webhook endpoint.
  4. Inspect the execution logs to determine whether the event was classified as fresh or stale.
  5. Inspect the generated record in the relevant DataGen task output.
  6. Send a payload without event_timestamp to test the warning path.

How to extend

  • Add email or Slack notifications when stale events are detected.
  • Send generated records to a database, queue, or monitoring platform.
  • Include producer identifiers, event types, or correlation IDs.
  • Adjust max_delay_minutes to suit different event-processing requirements.
  • Trigger downstream remediation workflows for stale events.

Links

See How

New to Kestra?

Use blueprints to kickstart your first workflows.