React Instantly to Kafka, SQS, and MQTT Events
For the complete documentation index, see llms.txt. For a full content snapshot, see llms-full.txt. Append.mdto anykestra.io/docs/*URL for plain Markdown.
Trigger workflows instantly as events occur, with millisecond latency.
Most Kestra triggers poll external systems at a fixed interval. Realtime Triggers listen directly for events and start a flow the moment one arrives, with millisecond latency. Each Realtime Trigger runs as a dedicated listener thread on a worker; when an event arrives, the listener immediately starts a workflow execution to process it.
Supported event sources
Realtime Triggers continuously listen for events and launch a new workflow execution the moment an event occurs, such as:
- a message is published to a Kafka topic
- a message is published to a Pulsar topic
- a message is published to an AMQP queue
- a message is published to an MQTT queue
- a message is published to an AWS SQS queue
- a message is published to Google Pub/Sub
- a message is published to Azure Event Hubs
- a message is published to a NATS subject
- an item is added to a Redis list
- a row is added, modified or deleted in Postgres, MySQL, or SQL Server.
Configuring a Realtime Trigger
A Realtime Trigger is configured by setting type to the plugin’s RealtimeTrigger class. The following example listens for new messages in an AWS SQS queue:
id: sqsnamespace: company.team
tasks: - id: log type: io.kestra.plugin.core.log.Log message: "{{ trigger }}"
triggers: - id: realtime_trigger type: io.kestra.plugin.aws.sqs.RealtimeTrigger region: eu-north-1 accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID')}}" secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}" queueUrl: https://sqs.eu-north-1.amazonaws.com/123456789/MyQueueWorker failover for Realtime Triggers
Each Realtime Trigger runs as a dedicated listener thread on one specific worker. If that worker stops, the listener stops with it. Kestra’s liveness mechanism detects this and re-emits the trigger so another available worker can pick it up.
The time before failover depends on how the worker stopped:
- Graceful shutdown (e.g.
docker stop, rolling deploy): the Executor waits forkestra.server.terminationGracePeriod(defaultPT5M) before reassigning the trigger. This prevents duplicate processing when the worker is expected to come back shortly, such as during a rolling deployment. - Abrupt failure (no heartbeat received): the Executor detects the missing heartbeat within
kestra.server.liveness.timeoutand reassigns the trigger without waiting for the grace period.
The failover time after a graceful shutdown is set by terminationGracePeriod:
kestra: server: terminationGracePeriod: PT1M # default is PT5MEvents are not lost during the failover window. They remain in the source system (Kafka topic, SQS queue, etc.) and will be consumed once the trigger listener is restarted on another worker.
Polling triggers vs. Realtime Triggers
| Criteria | Polling trigger | Realtime Trigger |
|---|---|---|
| Implementation | Micro-batch | Realtime |
| Event processing | Batch-processes all events received within the poll interval | Processes each event immediately as it arrives |
| Latency | Seconds or minutes | Milliseconds |
| Execution model | Each execution processes one or many events | Each execution processes exactly one event |
| Data handling | Stores all received events in a file | Stores each event in its raw format |
| Output format | URI of a file in internal storage | Raw event payload and metadata |
Realtime Triggers are stateless: each event creates its own independent workflow execution. They are designed for orchestrating workflows and microservices in response to events, not for continuous stateful stream processing.
To continue with Realtime Triggers, check out their How-to Guide.
Was this page helpful?