RealtimeTrigger
Trigger a flow on message consumption in real-time from Azure Event Hubs.
Trigger a flow on message consumption in real-time from Azure Event Hubs.
Trigger a flow on message consumption in real-time from Azure Event Hubs.
If you would like to consume multiple messages processed within a given time frame and process them in batch, you can use the io.kestra.plugin.azure.eventhubs.Trigger instead.
type: "io.kestra.plugin.azure.eventhubs.RealtimeTrigger"Examples
Trigger flow based on events received from Azure Event Hubs in real-time.
id: azure_eventhubs_realtime_trigger
namespace: company.team
tasks:
- id: log
type: io.kestra.plugin.core.log.Log
message: Hello there! I received {{ trigger.body }} from Azure EventHubs!
triggers:
- id: read_from_eventhub
type: io.kestra.plugin.azure.eventhubs.RealtimeTrigger
eventHubName: my_eventhub
namespace: my_eventhub_namespace
connectionString: "{{ secret('EVENTHUBS_CONNECTION') }}"
bodyDeserializer: JSON
consumerGroup: "$Default"
checkpointStoreProperties:
containerName: kestra
connectionString: "{{ secret('BLOB_CONNECTION') }}"
Use the Azure Event Hubs Realtime Trigger to push events into Azure Table Storage
id: eventhubs_realtime_trigger
namespace: company.team
tasks:
- id: insert_into_storagetable
type: io.kestra.plugin.azure.storage.table.Bulk
endpoint: https://yourstorageaccount.blob.core.windows.net
connectionString: "{{ secret('STORAGETABLE_CONNECTION') }}"
table: orders
from:
- partitionKey: order_id
rowKey: "{{ trigger.body | jq('.order_id') | first }}"
properties:
customer_name: "{{ trigger.body | jq('.customer_name') | first }}"
customer_email: "{{ trigger.body | jq('.customer_email') | first }}"
product_id: "{{ trigger.body | jq('.product_id') | first }}"
price: "{{ trigger.body | jq('.price') | first }}"
quantity: "{{ trigger.body | jq('.quantity') | first }}"
total: "{{ trigger.body | jq('.total') | first }}"
triggers:
- id: realtime_trigger
type: io.kestra.plugin.azure.eventhubs.RealtimeTrigger
eventHubName: orders
namespace: kestra
connectionString: "{{ secret('EVENTHUBS_CONNECTION') }}"
bodyDeserializer: JSON
consumerGroup: $Default
checkpointStoreProperties:
containerName: kestra
connectionString: "{{ secret('BLOB_CONNECTION') }}"
Properties
eventHubName*Requiredstring
The event hub to read from.
namespace*Requiredstring
Namespace name of the event hub to connect to.
bodyDeserializerstring
STRINGSTRINGBINARYIONJSONThe Deserializer to be used for serializing the event value.
bodyDeserializerPropertiesobject
{}The config properties to be passed to the Deserializer.
checkpointStorePropertiesobject
{}The config properties to be used for configuring the BlobCheckpointStore.
Azure Event Hubs Checkpoint Store can be used for storing checkpoints while processing events from Azure Event Hubs.
clientMaxRetriesintegerstring
5The maximum number of retry attempts before considering a client operation to have failed.
clientRetryDelayintegerstring
500The maximum permissible delay between retry attempts in milliseconds.
conditionsNon-dynamic
List of conditions in order to limit the flow trigger.
Condition to allow events between two specific datetime values.
date-timeThe date to test must be after this one.
date-timeThe date to test must be before this one.
Must be a valid ISO 8601 datetime with the zone identifier (use 'Z' for the default zone identifier).
{{ trigger.date }}Condition to allow events on a particular day of the week.
MONDAYTUESDAYWEDNESDAYTHURSDAYFRIDAYSATURDAYSUNDAY{{ trigger.date }}Condition to execute tasks on a specific day of the week relative to the current month (first, last, ...)
FIRSTLASTSECONDTHIRDFOURTHAre you looking for the first or the last day in the month?
MONDAYTUESDAYWEDNESDAYTHURSDAYFRIDAYSATURDAYSUNDAYThe day of week.
{{ trigger.date }}Condition for a specific flow of an execution.
Condition that checks labels of an execution.
List of labels to match in the execution.
Condition for an execution namespace.
String against which to match the execution namespace depending on the provided comparison.
EQUALSPREFIXSUFFIXComparison to use when checking if namespace matches. If not provided, it will use EQUALS by default.
falseWhether to look at the flow namespace by prefix. Shortcut for comparison: PREFIX.
Only used when comparison is not set
Condition based on the outputs of an upstream execution.
Condition based on execution status.
CREATEDSUBMITTEDRUNNINGPAUSEDRESTARTEDKILLINGSUCCESSWARNINGFAILEDKILLEDCANCELLEDQUEUEDRETRYINGRETRIEDSKIPPEDBREAKPOINTRESUBMITTEDCREATEDSUBMITTEDRUNNINGPAUSEDRESTARTEDKILLINGSUCCESSWARNINGFAILEDKILLEDCANCELLEDQUEUEDRETRYINGRETRIEDSKIPPEDBREAKPOINTRESUBMITTEDCondition based on variable expression.
Condition for a specific flow. Note that this condition is deprecated, use `io.kestra.plugin.core.condition.ExecutionFlow` instead.
The flow id.
The namespace of the flow.
Condition for a flow namespace.
The namespace of the flow or the prefix if prefix is true.
falseIf we must look at the flow namespace by prefix (checked using startsWith). The prefix is case sensitive.
Condition that matches if any taskRun has retry attempts.
CREATEDSUBMITTEDRUNNINGPAUSEDRESTARTEDKILLINGSUCCESSWARNINGFAILEDKILLEDCANCELLEDQUEUEDRETRYINGRETRIEDSKIPPEDBREAKPOINTRESUBMITTEDList of states that are authorized.
CREATEDSUBMITTEDRUNNINGPAUSEDRESTARTEDKILLINGSUCCESSWARNINGFAILEDKILLEDCANCELLEDQUEUEDRETRYINGRETRIEDSKIPPEDBREAKPOINTRESUBMITTEDList of states that aren't authorized.
Run a flow if the list of preconditions is met in a time window.
The list of preconditions to wait for
The key must be unique for a trigger because it will be used to store the previous evaluation result.
^[a-zA-Z0-9][a-zA-Z0-9_-]*1A unique id for the condition
trueWhether to reset the evaluation results of SLA conditions after a first successful evaluation within the given time period.
By default, after a successful evaluation of the set of SLA conditions, the evaluation result is reset, so, the same set of conditions needs to be successfully evaluated again in the same time period to trigger a new execution.
This means that to create multiple executions, the same set of conditions needs to be evaluated to true multiple times.
You can disable this by setting this property to false so that, within the same period, each time one of the conditions is satisfied again after a successful evaluation, it will trigger a new execution.
{
"type": "DURATION_WINDOW"
}Define the time period (or window) for evaluating preconditions.
You can set the type of sla to one of the following values:
DURATION_WINDOW: this is the defaulttype. It uses a start time (windowAdvance) and end time (window) that are moving forward to the next interval whenever the evaluation time reaches the end time, based on the defined durationwindow. For example, with a 1-day window (the default option:window: PT1D), the SLA conditions are always evaluated during 24h starting at midnight (i.e. at time 00: 00: 00) each day. If you setwindowAdvance: PT6H, the window will start at 6 AM each day. If you setwindowAdvance: PT6Hand you also override thewindowproperty toPT6H, the window will start at 6 AM and last for 6 hours — as a result, Kestra will check the SLA conditions during the following time periods: 06: 00 to 12: 00, 12: 00 to 18: 00, 18: 00 to 00: 00, and 00: 00 to 06: 00, and so on.SLIDING_WINDOW: this option also evaluates SLA conditions over a fixed timewindow, but it always goes backward from the current time. For example, a sliding window of 1 hour (window: PT1H) will evaluate executions for the past hour (so between now and one hour before now). It uses a default window of 1 day.DAILY_TIME_DEADLINE: this option declares that some SLA conditions should be met "before a specific time in a day". With the string propertydeadline, you can configure a daily cutoff for checking conditions. For example,deadline: "09: 00: 00"means that the defined SLA conditions should be met from midnight until 9 AM each day; otherwise, the flow will not be triggered.DAILY_TIME_WINDOW: this option declares that some SLA conditions should be met "within a given time range in a day". For example, a window fromstartTime: "06: 00: 00"toendTime: "09: 00: 00"evaluates executions within that interval each day. This option is particularly useful for declarative definition of freshness conditions when building data pipelines. For example, if you only need one successful execution within a given time range to guarantee that some data has been successfully refreshed in order for you to proceed with the next steps of your pipeline, this option can be more useful than a strict DAG-based approach. Usually, each failure in your flow would block the entire pipeline, whereas with this option, you can proceed with the next steps of the pipeline as soon as the data is successfully refreshed at least once within the given time range.
io.kestra.core.models.triggers.TimeWindow
partial-timeSLA daily deadline
Use it only for DAILY_TIME_DEADLINE SLA.
partial-timeSLA daily end time
partial-timeSLA daily start time
Use it only for DAILY_TIME_WINDOW SLA.
DURATION_WINDOWDAILY_TIME_DEADLINEDAILY_TIME_WINDOWDURATION_WINDOWSLIDING_WINDOWThe type of the SLA
The default SLA is a sliding window (DURATION_WINDOW) with a window of 24 hours.
durationUse it only for DURATION_WINDOW or SLIDING_WINDOW SLA.
See ISO_8601 Durations for more information of available duration value.
The start of the window is always based on midnight except if you set windowAdvance parameter. Eg if you have a 10 minutes (PT10M) window,
the first window will be 00: 00 to 00: 10 and a new window will be started each 10 minutes
durationUse it only for DURATION_WINDOW SLA.
Allow to specify the start time of the window
Eg: you want a window of 6 hours (window=PT6H), by default the check will be done between: 00: 00 and 06: 00, 06: 00 and 12: 00, 12: 00 and 18: 00, and 18: 00 and 00: 00.
If you want to check the window between 03: 00 and 09: 00, 09: 00 and 15: 00, 15: 00 and 21: 00, and 21: 00 and 3: 00, you will have to shift the window of 3 hours by settings windowAdvance: PT3H
durationThe duration of the window
Deprecated, use timeWindow.window instead.
durationThe window advance duration
Deprecated, use timeWindow.windowAdvance instead.
Condition to exclude other conditions.
1The list of conditions to exclude.
If any condition is true, it will prevent the event's execution.
Condition to have at least one condition validated.
1The list of conditions to validate.
If any condition is true, it will allow the event's execution.
Condition to allow events on public holidays.
ISO 3166-1 alpha-2 country code. If not set, it uses the country code from the default locale.
{{ trigger.date}}ISO 3166-2 country subdivision (e.g., provinces and states) code.
It uses the Jollyday library for public holiday calendar that supports more than 70 countries.
Condition to allow events between two specific times.
timeThe time to test must be after this one.
timeThe time to test must be before this one.
Must be a valid ISO 8601 time with offset.
{{ trigger.date }}The time to test.
Can be any variable or any valid ISO 8601 time. By default, it will use the trigger date.
Condition to allow events on weekend.
{{ trigger.date }}The date to test.
Can be any variable or any valid ISO 8601 datetime. By default, it will use the trigger date.
connectionStringstring
Connection string of the Storage Account.
consumerGroupstring
$DefaultThe consumer group.
customEndpointAddressstring
Custom endpoint address when connecting to the Event Hubs service.
enqueueTimestring
The ISO Datetime to be used when PartitionStartingPosition is configured to INSTANT.
Configs in key/value pairs.
partitionStartingPositionstring
EARLIESTEARLIESTLATESTINSTANTThe starting position.
sasTokenstring
The SAS token to use for authenticating requests.
This string should only be the query parameters (with or without a leading '?') and not a full URL.
stopAfterNon-dynamicarray
CREATEDSUBMITTEDRUNNINGPAUSEDRESTARTEDKILLINGSUCCESSWARNINGFAILEDKILLEDCANCELLEDQUEUEDRETRYINGRETRIEDSKIPPEDBREAKPOINTRESUBMITTEDList of execution states after which a trigger should be stopped (a.k.a. disabled).
