Available on: >= 0.17.0

React to events as they happen with millisecond latency.

Triggers in Kestra can listen to external events and start a workflow execution when the event occurs. Most of these triggers poll external systems for new events at regular intervals e.g. every second. This works well for data processing use cases. However, business-critical workflows often require reacting to events as they happen with millisecond latency and this is where Realtime Triggers come into play.

What are Realtime Triggers

Realtime triggers listen to events in real time and start a workflow execution as soon as:

How Realtime Triggers Work

As soon as you add a Realtime Trigger to your workflow, Kestra starts an always-on thread that listens to the external system for new events. When a new event occurs, Kestra starts a workflow execution to process the event.

Use Cases

Using Realtime Triggers, you can orchestrate business-critical processes and microservices in real time. This covers scenarios such as:

  • fraud and anomaly detection,
  • order processing,
  • realtime predictions or recommendations,
  • reacting to stock price changes,
  • shipping and delivery notifications,
  • ...and many more use cases that require reacting to events as they happen.

In addition, Realtime Triggers can be used for data orchestration, especially for Change Data Capture use cases. The Debezium Postgres RealtimeTrigger plugin can listen to changes in a database table and start a workflow execution as soon as a new row is inserted, updated, or deleted.

When to use Triggers vs. Realtime Triggers

The table below compares Triggers with Realtime Triggers to help you choose the right trigger type for your use case:

CriteriaTriggerRealtime Trigger
Event ProcessingBatch-process all events received until the poll interval has elapsedProcess each event immediately as it happens
LatencySecond(s) or minute(s)Millisecond(s)
Execution ModelEach execution processes one or many eventsEach execution processes exactly one event
Data HandlingStore all received events in a fileStore each event in a raw format
Output formatURI of a file in internal storageRaw data of the event payload and related metadata
ApplicationData applications processing data in batchBusiness-critical operations reacting to events in real time
Use casesData orchestration for analytics and building data productsProcess and microservice orchestration (real time updates, anomaly detection, order processing)

How to Use Realtime Triggers

To use Realtime Triggers, simply choose the RealtimeTrigger as a trigger type of your desired service. Here, we use the RealtimeTrigger to listen to new messages in an AWS SQS queue:

id: sqs
namespace: company.team

  - id: log
    type: io.kestra.plugin.core.log.Log
    message: "{{ trigger }}"

  - id: realtime_trigger
    type: io.kestra.plugin.aws.sqs.RealtimeTrigger
    region: eu-north-1
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID')}}"
    secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"
    queueUrl: https://sqs.eu-north-1.amazonaws.com/123456789/MyQueue

Comparison with Realtime Data Processing Engines

It's important to note that Kestra's Realtime Triggers are not intended to be used as a replacement for real-time data processing engines such as Apache Flink, Apache Beam or Google Dataflow.

Those data processing engines excel at stateful streaming applications and complex SQL transformations over real-time data streams.

In contrast, Kestra's Realtime Triggers are stateless, meaning they trigger one workflow execution per event. They are designed primarily to react to events in real time to orchestrate business-critical processes.

Was this page helpful?