Monitor Azure Event Hub Volume and Alert on a Silent Drop

Catch a dead Event Hub producer before your downstream consumers notice the gap. This blueprint polls an Azure Event Hub every 5 minutes, counts how many events arrived in a 30-second consumption window, and pages on-call the moment that count falls below the floor you expect for a healthy producer. It closes the common gap in Azure Event Hub orchestration: nothing in Event Hubs itself tells you when a stream goes quiet, so a stalled producer looks identical to a slow news day until a downstream job starts failing on missing data.

How it works

• A io.kestra.plugin.core.trigger.Schedule fires every 5 minutes (*/5 * * * *).
• io.kestra.plugin.azure.eventhubs.Consume polls the telemetry hub in the kestra-eventhub-ns namespace for up to PT30S, deserializes bodies as JSON, and checkpoints progress to a Blob Storage container so the next run picks up where this one left off.
• io.kestra.plugin.core.flow.If evaluates outputs.consume_batch.eventsCount < 5. Tune this floor to whatever your quietest healthy window looks like.
• When the condition is true, io.kestra.plugin.pagerduty.PagerDutyAlert fires a trigger event with a dedup_key, so repeated low-volume windows update the same incident instead of opening a new one every 5 minutes.

What you get

• Automatic detection of a silent Event Hub producer, no dashboard-watching required.
• Checkpointed consumption, so the monitor never re-reads events it already counted.
• A single deduplicated PagerDuty incident instead of alert spam on every poll.
• A pattern that adapts to any Event Hub by swapping the hub name and the volume floor.

Who it's for

• Platform and SRE teams responsible for streaming pipeline reliability.
• Data engineers who need to know when an upstream IoT or app-telemetry producer goes dark.
• Teams migrating from a custom watchdog script to a declarative, versioned flow.

Why orchestrate this with Kestra

Event Hubs has no native "alert me when volume drops" primitive; that logic usually lives in a bespoke Azure Function on a timer, with its own retry and alerting code to maintain. Kestra replaces that custom function with a declarative flow: the schedule, the consumption window, the threshold check, and the PagerDuty call are all one auditable execution history, with retries and checkpointing handled by the platform instead of hand-rolled code.

Prerequisites

• An Azure Event Hubs namespace and hub to monitor.
• A Blob Storage container for Event Hub checkpoints.
• A PagerDuty Events API v2 integration.

Secrets

• EVENTHUBS_CONNECTION: Event Hubs connection string with listen permissions.
• BLOB_CONNECTION: Storage account connection string for the checkpoint container.
• PAGERDUTY_EVENT_URL: PagerDuty Events API v2 endpoint (https://events.pagerduty.com/v2/enqueue).
• PAGERDUTY_ROUTING_KEY: PagerDuty integration routing key.

Quick start

1. Add the four secrets above to your Kestra instance.
2. Set eventHubName and namespace to your Event Hub.
3. Set checkpointStoreProperties.containerName to an existing Blob container.
4. Tune the eventsCount < 5 floor to your producer's normal quiet-window volume.
5. Deploy the flow and watch it page on the next silent window.

How to extend

• Swap PagerDutyAlert for io.kestra.plugin.slack.notifications.SlackIncomingWebhook or Microsoft Teams if PagerDuty isn't in your stack.
• Add an upper-bound check alongside the floor to catch an unexpected traffic spike too.
• Chain a downstream step that writes the count to Azure Monitor for a historical volume chart.
• Replace the fixed floor with a rolling average computed from a prior task's output.

Links

• Plugin docs: Azure
• Orchestrate Azure with Kestra