Ingest a Documentation Site into Meilisearch with Python Shaping

Give your documentation site a fast search index that refreshes itself. This blueprint chains an HTTP download, a Python shaping step, and io.kestra.plugin.meilisearch.DocumentAdd into one pipeline: the raw JSON export of the site lands in Kestra's internal storage, Python strips every record down to the id, title, and body fields the index actually needs, and Meilisearch ingests the batch while the task waits for indexing to complete. A Slack message closes the loop with the exact document count, so the content team sees every refresh.

How it works

1. download_content (io.kestra.plugin.core.http.Download) fetches the JSON content dump and stores it in internal storage, exposed as {{ outputs.download_content.uri }}.
2. shape_documents (io.kestra.plugin.scripts.python.Script on the Process task runner, no container image required) reads the raw file, writes one lean search document per line to docs.ion, and emits doc_count through Kestra's output protocol, addressable as {{ outputs.shape_documents.vars.doc_count }}.
3. index_documents (io.kestra.plugin.meilisearch.DocumentAdd) sends the shaped file to the docs_site index in batches of 500. With waitForIndexing: true and a PT5M timeout, the task only succeeds once Meilisearch reports the enqueued indexing tasks as done, and its documentsAdded output carries the indexed count.
4. notify posts both counts to Slack; the errors block posts a distinct alert when any step fails.
5. A disabled-by-default Schedule trigger refreshes the index daily at 05:00.

What you get

• A complete content-to-search pipeline in four tasks, from raw JSON to a queryable Meilisearch index.
• Shaping in plain Python on the host, no container image to build or pull.
• A green execution that guarantees searchability, because the flow waits for Meilisearch indexing instead of just enqueueing it.
• Document counts as first-class outputs, ready for notifications or downstream checks.

Who it's for

• Documentation and content teams who want site search that stays in sync with published content.
• Data engineers replacing ad hoc reindex scripts with an observable, scheduled pipeline.
• Anyone evaluating Meilisearch who wants a working ingest in one copy-paste.

Why orchestrate this with Kestra

Meilisearch indexes documents, but something still has to fetch the content on schedule, transform it consistently, wait for indexing to finish, and tell someone when the refresh breaks. Kestra provides all of that around four declarative tasks: internal storage carries the file between steps, the output protocol turns script results into template variables, and every refresh is auditable in the execution history.

Prerequisites

• A running Meilisearch instance reachable from Kestra.
• A JSON endpoint or export of your content; the demo uses a public placeholder API with id, title, and body fields.
• A Slack incoming webhook for confirmations and failure alerts.

Secrets

• MEILISEARCH_URL: Meilisearch connection URL, e.g. https://meilisearch.internal:7700.
• MEILISEARCH_KEY: Meilisearch API key with document write permissions.
• SLACK_WEBHOOK_URL: Slack incoming webhook URL.

Quick start

1. Add the three secrets to your Kestra namespace.
2. Execute the flow and check that Slack reports the shaped and indexed counts.
3. Point download_content at your real content export and adjust the field mapping in shape_documents.
4. Set disabled: false on the daily trigger.

How to extend

• Upsert a sentinel document with an updated_at timestamp in the shaping step, so the freshness probe blueprint can verify the pipeline keeps running.
• Split large corpora across several DocumentAdd tasks per section, each targeting its own index.
• Add a Search task after indexing as a smoke test for a known query, as shown in the search canary blueprint.
• Enrich documents in the Python step with section labels to power faceted search, as shown in the facet report blueprint.

Links

• Meilisearch plugin documentation
• Meilisearch documentation