Matrix Workflow Failure Alert Dispatcher with KV Store Deduplication and Multi-Room Routing

When automated data pipelines or microservice workflows fail across multiple environments, operational teams face alerting noise and delayed response times. Sending an instant message for every repeated failure leads to alert fatigue, while sending all alerts to a single chat channel obscures critical production outages among routine development warnings.

This operational blueprint provides a centralized Matrix incident dispatcher for Kestra. It monitors workflow executions, deduplicates repeated alerts using Kestra KV store state, and dynamically routes failure notifications to environment-specific Matrix rooms (#prod-incidents, #dev-alerts, #general-alerts).

How it works

1. Execution Event Trigger: The on_failure trigger (io.kestra.plugin.core.trigger.Flow) listens for FAILED or WARNING execution states across all company.* namespaces. The condition trigger.flowId != flow.id prevents self-triggering loops.
2. KV Deduplication Check: Task check_recent_alert (io.kestra.plugin.core.kv.Get) looks up the sanitized key dedup_{namespace}_{flowId} in Kestra KV store with errorOnMissing: false.
3. Conditional Execution: Task evaluate_alert (io.kestra.plugin.core.flow.If) evaluates outputs.check_recent_alert.value == null:
   • If No Active Cooldown:
     • Task route_by_namespace (io.kestra.plugin.core.flow.Switch) performs exact string matching on trigger.namespace (e.g. company.prod, company.dev). Any namespace not explicitly matched falls back to defaults and dispatches to default_room_id.
     • Sends a plain text NOTICE message to the designated Matrix room using io.kestra.plugin.matrix.Send and secret MATRIX_ACCESS_TOKEN.
     • Task record_alert_cooldown (io.kestra.plugin.core.kv.Set) writes the deduplication key with a TTL duration specified by inputs.cooldown (default PT15M).
   • If Cooldown Active:
     • Task log_suppressed_alert (io.kestra.plugin.core.log.Log) logs a suppression notice and skips sending a chat message.
4. Failure Isolation: An errors block logs dispatcher runtime issues without calling external chat endpoints to avoid circular notification loops.

What you get

• Centralized, hands-off failure alert routing across all workflows in watched namespaces.
• Automatic alert deduplication preventing chat spam during retry loops.
• Multi-room environment routing separating critical production outages from development warnings.

Who it's for

• SREs and DevOps Engineers: Who manage multi-environment Kestra deployments and use Matrix for real-time operations chat.
• Platform Engineering Teams: Looking for a self-hosted, open-source alternative to proprietary incident alerting platforms.

Why orchestrate

Centralizing failure dispatching inside Kestra eliminates redundant webhook configurations in individual workflows. Using a Flow trigger combined with KV state deduplication ensures that temporary workflow retry loops or cascading job failures do not spam chat channels.

How to extend

• Add additional namespace branches (e.g. company.staging, company.data) to the Switch task to route to dedicated team rooms.
• Integrate PagerDuty or webhook tasks alongside Matrix for high-priority production outages.
• Adjust the cooldown duration input per environment to customize alert suppression windows.

Prerequisites

• A running Matrix homeserver (e.g., https://matrix.org or a self-hosted Synapse server).
• A dedicated Matrix bot account.
• Target Matrix rooms created and configured as UNENCRYPTED rooms.
• The bot account must be invited to each target room and joined prior to flow execution.
• Note on Inputs: Because event-triggered flows do not accept runtime input prompts when executed by triggers, customize your Matrix homeserver URL, room IDs, and cooldown duration directly in the flow's inputs defaults.

Secrets

• MATRIX_ACCESS_TOKEN: Access token for the Matrix bot account.
  • How to retrieve: In Element desktop/web client, navigate to Settings > Help & About > Advanced > Access Token. (Other token retrieval methods like API calls are marked UNVERIFIED).

Quick start

1. Add the secret MATRIX_ACCESS_TOKEN to your Kestra namespace.

2. Create an unencrypted room in Matrix (e.g., #prod-incidents), invite your bot, and copy the internal room ID (e.g., !abc123xyz:matrix.org from Room Settings > Advanced).

3. Import this blueprint and set prod_room_id, dev_room_id, and default_room_id input defaults.

4. Test the dispatcher by executing this sample failing flow in a watched namespace:

   id: sample-failing-job
   namespace: company.prod
   tasks:
     - id: trigger_failure
       type: io.kestra.plugin.core.execution.Fail
       message: "Simulated production pipeline failure for Matrix alert test."
   

5. Execute sample-failing-job. The dispatcher will catch the failure, post a notice to Matrix, and set a 15-minute KV deduplication lock.

Expected output

• check_recent_alert returns null on first run.
• Matrix receives a plain text NOTICE message with direct execution link ({{ kestra.url }}/ui/executions/...).
• KV store sets key dedup_company_prod_sample-failing-job with PT15M expiration.
• Re-executing sample-failing-job within 15 minutes logs "Alert for flow sample-failing-job in namespace company.prod was suppressed".

Pitfalls

• Manual Execution Behavior: Executing the dispatcher flow manually from the Kestra UI will fail because trigger.* context variables (trigger.namespace, trigger.flowId, trigger.executionId, trigger.state) are only populated during event-driven trigger executions. Always test by executing a sample failing flow.
• End-to-End Encryption (E2EE): Matrix rooms must be unencrypted. Kestra's Matrix plugin does not support Olm/Megolm encryption ratchets; posting to E2EE rooms will fail.
• Bot Room Membership: The bot must be joined to the room before execution. Uninvited or unjoined rooms return HTTP 403 M_FORBIDDEN.
• Rate Limits: Homeservers enforce rate limits on message events. KV deduplication protects homeservers from spam.
• Room Aliases: Use internal room IDs (!id:domain), not room aliases starting with #.

Links

• Kestra Matrix Plugin Documentation
• Kestra Core KV Store Documentation
• Kestra Flow Triggers Documentation
• This blueprint has been created by mohitjeswani01