Run Metaplane Monitors Right After a Load

Scheduled monitor sweeps have a blind spot, the gap between a load finishing and the next sweep starting. If the load lands at 02:25 and the monitors run at 06:00, a broken load sits undetected for hours. This blueprint closes the gap with io.kestra.plugin.metaplane.Run: the moment the load task succeeds, Kestra enqueues an immediate run of the monitors covering the loaded tables. The call is fire-and-forget, the Metaplane API answers as soon as the run is enqueued, so the flow finishes in seconds while detection happens on Metaplane's side.

How it works

1. load_data stands in for the real ingestion step, a dlt pipeline, an Airbyte sync, or a warehouse COPY. Because run_monitors only executes after it succeeds, monitors always sweep post-load data, never a half-written table.
2. run_monitors (io.kestra.plugin.metaplane.Run) posts the monitorIds to Metaplane in one call. The task returns the enqueued monitor IDs as its output and does not wait for the monitors to complete.
3. note_enqueued posts a Discord note with the count of enqueued monitors, computed from the task output, so the channel knows fresh results are on the way.
4. The errors block sends a distinct alert when the load or the enqueue fails, because a silently skipped sweep means anomalies wait for the next scheduled run.
5. A disabled-by-default Schedule trigger runs the flow after the nightly load window.

What you get

• Monitor results that track the warehouse instead of the clock, freshness checks run when data actually changes.
• A flow that finishes in seconds, since enqueueing is asynchronous and nothing polls for results.
• Load and observability coupled in one execution, visible together in Kestra's execution history.
• A pattern that scales to any number of monitors by extending one list.

Who it's for

• Data engineers whose loads finish at varying times, making fixed monitor schedules either late or wasteful.
• Teams using Metaplane freshness and volume monitors who want detection latency measured in minutes, not hours.
• Anyone building toward gated publishes, fresh results make a later Gate task resolve quickly.

Why orchestrate this with Kestra

Only the orchestrator knows the exact moment a load finished. Kestra turns that knowledge into action: the Run task fires as the next step of the same execution, inherits the flow's error handling, and leaves an audit trail tying each load to the sweep it triggered. When the sweep should block rather than inform, the same plugin's Gate task swaps in without changing the flow's shape.

Prerequisites

• A Metaplane workspace with monitors on the tables your load writes, and their monitor UUIDs.
• A Metaplane API token.
• A Discord incoming webhook for the sweep notes.

Secrets

• METAPLANE_API_TOKEN: Metaplane API token.
• DISCORD_WEBHOOK_URL: Discord incoming webhook URL.

Quick start

1. Add both secrets to your Kestra namespace.
2. Replace the placeholder monitorIds with UUIDs from your Metaplane workspace.
3. Execute the flow and confirm the Discord note reports the enqueued count, then check Metaplane for the fresh results.
4. Replace the load_data stand-in with your real ingestion task and enable the after_nightly_load trigger.

How to extend

• Replace the stand-in with io.kestra.plugin.dlt.Run, an Airbyte sync task, or a dbt build so the sweep follows your real load.
• Upgrade to io.kestra.plugin.metaplane.Gate with runFirst: true when downstream steps should wait for the results instead of firing and forgetting, as shown in the quality gate blueprint.
• Trigger the flow from a Kestra flow trigger on the ingestion flow's SUCCESS state instead of a schedule, so any load anywhere refreshes its monitors.
• Add a second Run task with a different monitor list for tables loaded by a separate pipeline.

Links

• Metaplane plugin documentation
• Metaplane documentation