Segment Reverse ETL Delivery Failure Gate

A Segment Reverse ETL sync can report an overall SUCCESS while individual records still fail to land in the destination, rate limited, rejected by a validation rule, or otherwise dropped. Segment's own sync history shows this if someone opens the right sync and reads the load phase counts; it pages nobody. This blueprint triggers the sync, waits for it to finish, and treats a nonzero delivery failure count as seriously as an outright failed sync.

This blueprint was created by Mohit Jeswani.

How it works

1. run_sync (io.kestra.plugin.twilio.segment.reverseetl.Sync, wait: true) triggers a manual sync for model_id/subscription_id and polls until it reaches SUCCESS or FAIL. errorOnFailing: false keeps the task returning its Output either way, so evaluate controls the classification instead of the task throwing on an outright failure. allowFailure: true covers an API or timeout error without failing the flow outright.
2. evaluate (io.kestra.plugin.core.output.OutputValues) reads the sync's own status and loadPhase.deliverFailureCount. FAILED when the top-level status is not SUCCESS; PARTIAL_FAILURE when the status is SUCCESS but deliverFailureCount still reaches max_delivery_failures; OK otherwise. A missing result entirely is CRITICAL.
3. route (io.kestra.plugin.core.flow.If):
   • compliant: logs a clean sync.
   • alert: posts the status and delivery failure count to Slack.
4. summary logs the status, delivery failure count, and severity, closing the governance cycle.
5. errors alerts Slack if the flow itself fails outside the handled cases.

What you get

• A delivery failure count checked on every sync, not just whether Segment's own top-level status says SUCCESS, which a partial failure can satisfy while still dropping records.
• Outright failure and partial delivery failure treated as distinct findings in the Slack alert text, not collapsed into one generic "something went wrong."
• A full execution history of every sync's outcome, not just whatever the last person to check Segment's own sync history happened to notice.

Who it's for

• Data platform teams running Segment Reverse ETL who need the destination to actually receive every record, not just a green checkmark on the sync itself.
• Teams evaluating the Twilio Segment plugin who want a worked example of reverseetl.Sync used as a monitored gate rather than a plain fire-and-forget trigger.

Why orchestrate this with Kestra

Triggering a Reverse ETL sync from Segment's own UI or API answers one moment in time; it does not decide a cadence, read the load-phase delivery counts, or notify anyone when some records still failed inside an overall-successful sync. Kestra supplies the schedule, the OK/PARTIAL_FAILURE/FAILED classification as a first-class branch, and an execution history that shows exactly when a sync started dropping records.

Prerequisites

• A Segment workspace with a Reverse ETL model, subscription (mapping), and warehouse source already configured.
• A Segment API token scoped for Reverse ETL.
• A Slack incoming webhook URL.

This flow calls the Segment Public API directly over HTTPS; there is no official local Segment emulator, so run_sync cannot be exercised against a local Docker container, and it genuinely triggers a real sync on every run. Validate against a non-production model first.

Secrets

• SEGMENT_TOKEN: the Segment API token used by run_sync. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_SEGMENT_TOKEN.
• SLACK_WEBHOOK_URL: used by alert and the errors block. In Kestra OSS, provide it as the base64-encoded environment variable SECRET_SLACK_WEBHOOK_URL.

Inputs

Input	Default	Purpose
source_id	<your-segment-source-id>	Warehouse source backing the sync
model_id	<your-segment-model-id>	Reverse ETL model to sync
subscription_id	<your-segment-subscription-id>	Subscription (mapping) to run
max_duration	PT30M	Ceiling this task waits for a terminal status
max_delivery_failures	1	deliverFailureCount at or above this is PARTIAL_FAILURE

Outputs

• {{ outputs.run_sync.syncId }}: the triggered sync's ID.
• {{ outputs.run_sync.status.loadPhase.deliverFailureCount }} / .deliverSuccessCount: delivery outcome counts for the sync.
• {{ outputs.evaluate.values.severity }}: OK, PARTIAL_FAILURE, FAILED, or CRITICAL.

Pitfalls

• This flow triggers a real sync every time it runs. It is not a passive status poll; do not point model_id/subscription_id at a sync where an extra, unplanned run is unsafe or expensive for the destination system.
• errorOnFailing: false is deliberate here, not an oversight: the task's own failure handling only recognizes the top-level status, not deliverFailureCount, so letting the task throw would have missed the partial-failure case entirely.
• deliverFailureCount and the other load/extract phase fields are returned as strings by Segment's API; the comparison against max_delivery_failures coerces the value with Pebble's number filter.
• A timeout past max_duration is treated the same as any other failure to return a result (via allowFailure); it is not distinguished from an outright FAIL status in the Slack alert text.

Quick start

1. Add the SEGMENT_TOKEN and SLACK_WEBHOOK_URL secrets.
2. Set source_id, model_id, and subscription_id to a real, non-production sync.
3. Run the flow manually and confirm compliant logs a clean sync.
4. If you can reproduce a partial delivery failure in a test destination, confirm alert fires with PARTIAL_FAILURE and the real failure count.
5. Enable the hourly trigger only once you are certain a sync on every run is acceptable for this model.

How to extend

• Loop over several model_id/subscription_id pairs with io.kestra.plugin.core.flow.Loop to watch every Reverse ETL mapping in one scheduled run, with concurrencyLimit: 1 to avoid triggering every sync at once.
• Read outputs.run_sync.status.extractPhase alongside loadPhase to distinguish a source extraction problem from a destination delivery problem.
• Chain a downstream task that re-queues just the failed records once Segment exposes a per-record failure export, rather than re-running the whole model.

Links

• Twilio Segment plugin: https://kestra.io/plugins/plugin-twilio
• Segment Reverse ETL API reference: https://segment.com/docs/api/public-api/
• Slack incoming webhooks: https://api.slack.com/messaging/webhooks