Rebuild a Typesense Product Search Index Nightly with Bulk Indexing

Product search is only as good as the index behind it. This blueprint is the nightly rebuild for an e-commerce catalog: io.kestra.plugin.core.http.Download pulls the product feed from an API, a small Python script reshapes it into one JSON document per line, and io.kestra.plugin.typesense.BulkIndex streams the file into the products collection in batches of 100 documents. Slack gets the converted and indexed counts after every run, so a shrinking catalog is visible the morning it happens.

How it works

1. download_catalog (io.kestra.plugin.core.http.Download) fetches the product catalog JSON into Kestra's internal storage. The demo uses a public dummy products API returning 100 products.
2. convert_to_jsonl (io.kestra.plugin.scripts.python.Script on the Process task runner, standard library only) maps each product to the collection schema, writes products.jsonl as an output file, and emits doc_count through Kestra's output protocol, addressable as {{ outputs.convert_to_jsonl.vars.doc_count }}.
3. index_catalog (io.kestra.plugin.typesense.BulkIndex) reads the JSONL file from internal storage via from and indexes it in chunk: 100 batches. The task exposes {{ outputs.index_catalog.size }}, the total number of documents sent to Typesense.
4. notify posts both counts to Slack; the errors block posts a distinct alert naming the flow and execution when any step fails.
5. A disabled-by-default Schedule trigger rebuilds the index nightly at 02:00.

The connection uses the Typesense default port 8108 as a plain value and https: false for a local or private-network cluster; set https: true when your production cluster terminates TLS.

What you get

• A complete catalog-to-search pipeline in four tasks, with no custom client code for Typesense.
• Chunked bulk indexing that keeps memory flat on large catalogs; tune chunk to your cluster.
• The converted and indexed document counts as first-class outputs, ready for alerting thresholds.
• A Slack trail for every rebuild, plus a failure alert so an empty index never goes unnoticed.

Who it's for

• E-commerce engineers keeping a storefront search index in sync with the product catalog.
• Search teams replacing hand-rolled indexing scripts with an observable, scheduled pipeline.
• Anyone evaluating Typesense who wants a working end-to-end indexing flow in one copy-paste.

Why orchestrate this with Kestra

A rebuild script on cron tells nobody when it half-finishes. Kestra runs the same three steps with an execution history, passes the downloaded file and the JSONL through internal storage instead of a shared disk, surfaces document counts as outputs that flow straight into Slack, and raises a distinct alert on failure. Swapping the source API or the collection is a one-line change in YAML.

Prerequisites

• A running Typesense cluster reachable from Kestra.
• The target collection must exist with its schema before the first run; BulkIndex indexes documents, it does not create collections. The Quick start below includes the curl call.
• A Slack incoming webhook for rebuild reports.

Secrets

• TYPESENSE_HOST: hostname or IP of the Typesense node or load balancer.
• TYPESENSE_API_KEY: an admin API key with write access to the collection.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Create the collection with its schema (adjust host and key):

   curl -X POST "http://localhost:8108/collections" \
     -H "X-TYPESENSE-API-KEY: $TYPESENSE_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{
       "name": "products",
       "fields": [
         {"name": "name", "type": "string"},
         {"name": "description", "type": "string"},
         {"name": "brand", "type": "string", "facet": true},
         {"name": "category", "type": "string", "facet": true},
         {"name": "price", "type": "float"},
         {"name": "popularity", "type": "float"},
         {"name": "in_stock", "type": "bool", "facet": true}
       ]
     }'
   

2. Add the three secrets to your Kestra namespace.

3. Execute the flow and check the Slack message reports 100 converted and 100 indexed documents.

4. Point download_catalog at your real product feed, adjust the Python mapping, and set disabled: false on the nightly trigger.

How to extend

• Add a sentinel document write after index_catalog and verify it with the Typesense rebuild canary blueprint, which reads it back with DocumentGet.
• Guard relevance after each rebuild with the Typesense search canary blueprint.
• Raise chunk for larger catalogs, or lower it if your cluster is memory constrained.
• Swap the dummy API for your PIM export and pass authentication headers on the Download task from {{ secret('...') }}.

Links

• Typesense plugin documentation
• Typesense documentation