Typesense Rebuild Canary Reading a Sentinel Document

A green rebuild execution proves the pipeline ran; it does not prove the index changed. This blueprint closes that gap with a sentinel: the rebuild flow writes one extra document, catalog-sentinel, carrying a rebuilt_at epoch timestamp as its final step. Thirty minutes after the rebuild window, io.kestra.plugin.typesense.DocumentGet reads the sentinel back from the same cluster the storefront queries, a Python task compares rebuilt_at against the clock, and a stale or missing sentinel pages Discord and fails the execution.

How it works

1. get_sentinel (io.kestra.plugin.typesense.DocumentGet) fetches document id catalog-sentinel from the products collection and exposes it as the document map output. If the document does not exist, the task fails and the errors block raises the alert, covering the missing case.
2. check_freshness (io.kestra.plugin.scripts.python.Script on the Process task runner) receives the document as JSON through an env variable rendered with {{ outputs.get_sentinel.document | toJson }}, computes the age in hours, and emits stale and age_hours through Kestra's output protocol. The threshold is 26 hours, one nightly cycle plus slack.
3. verdict (io.kestra.plugin.core.flow.If) checks {{ outputs.check_freshness.vars.stale }}. When stale, Discord gets the sentinel age and io.kestra.plugin.core.execution.Fail marks the execution failed, so the missed rebuild surfaces as an incident.
4. log_fresh records the age on healthy runs, building a history of how quickly rebuilds land.
5. A disabled-by-default Schedule trigger runs at 02:30, 30 minutes after the nightly catalog index blueprint's 02:00 rebuild; align the two crons if you change either.

The connection uses the Typesense default port 8108 as a plain value and https: false for a local or private-network cluster; set https: true when your production cluster terminates TLS.

What you get

• End-to-end proof that the rebuild's documents reached the serving cluster, not just that the pipeline exited zero.
• Three distinguishable outcomes: fresh sentinel, stale sentinel, and missing sentinel or unreachable cluster.
• The sentinel age in hours as a scalar output on every run.
• A failed execution on staleness, so existing failure-based alerting picks it up with zero extra wiring.

Who it's for

• E-commerce teams who have been burned by a green pipeline and a day-old index.
• Search engineers verifying that reindexing jobs land on the cluster customers actually query.
• SREs who prefer read-back verification over trusting a writer's own success report.

Why orchestrate this with Kestra

The sentinel pattern needs a writer, a reader on a related schedule, a comparison, and an escalation path. Kestra hosts all four: the rebuild flow writes the sentinel as its last task, this flow reads it back on an offset cron, the Python check runs on the built-in Process runner with no dependencies, and the verdict flows into Discord and execution state. Both flows live side by side in the same namespace with full execution history.

Prerequisites

• The nightly catalog index blueprint, or any rebuild job, writing the sentinel as its final step; a DocumentIndex task with document: {id: catalog-sentinel, rebuilt_at: ...} does it in four lines.
• The products collection schema must include an optional rebuilt_at int64 field; the Quick start adds it.
• A Discord incoming webhook for alerts.

Secrets

• TYPESENSE_HOST: hostname or IP of the Typesense node or load balancer.
• TYPESENSE_API_KEY: an API key with read access to the collection.
• DISCORD_WEBHOOK_URL: Discord incoming webhook URL.

Quick start

1. Add the rebuilt_at field to the collection and write an initial sentinel (adjust host and key):

   curl -X PATCH "http://localhost:8108/collections/products" \
     -H "X-TYPESENSE-API-KEY: $TYPESENSE_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"fields": [{"name": "rebuilt_at", "type": "int64", "optional": true}]}'
   
   curl -X POST "http://localhost:8108/collections/products/documents?action=upsert" \
     -H "X-TYPESENSE-API-KEY: $TYPESENSE_API_KEY" \
     -H "Content-Type: application/json" \
     -d "{\"id\": \"catalog-sentinel\", \"name\": \"catalog sentinel\", \"description\": \"rebuild marker\", \"brand\": \"internal\", \"category\": \"internal\", \"price\": 0.0, \"popularity\": 0.0, \"in_stock\": false, \"rebuilt_at\": $(date +%s)}"
   

2. Add the three secrets to your Kestra namespace and execute the flow; the healthy log line reports the sentinel age.

3. Wait 26 hours or upsert a sentinel with an old rebuilt_at to see the Discord alert and the failed execution.

4. Add the sentinel upsert as the last task of your rebuild flow and set disabled: false on the after_rebuild trigger.

How to extend

• Write the sentinel from the rebuild flow with io.kestra.plugin.typesense.DocumentIndex and rebuilt_at: "{{ execution.startDate | timestamp }}".
• Tighten the threshold from 26 hours to match your rebuild cadence, or parameterize it as a flow input.
• Add a second sentinel per collection when several indexes rebuild on different schedules.
• Chain this flow directly after the rebuild with a Flow trigger instead of the offset cron.

Links

• Typesense plugin documentation
• Typesense documentation