Schedule icon
Request icon
Return icon
Switch icon
Log icon
SlackIncomingWebhook icon

Replicate Prediction Failure Sentinel

Read recent Replicate predictions every hour, run no model, and alert Slack when failures cross your limits, with the failing model and top error.

Categories
AI

Image generation, transcription, upscaling and LLM features built on Replicate fail one prediction at a time: a model owner pushes a new version, an input field changes type, a GPU runs out of memory on large inputs, or the account hits a spend limit. Each prediction just ends with status: failed and an error string, the app shows a generic error, and nobody notices the rate climbing until users complain. This blueprint turns a scheduled, read-only call to the Replicate API into an inference health gate: io.kestra.plugin.core.http.Request reads recent predictions, one jq query computes the failure rate, the model that fails most and the most common error, and the result routes to a quiet log or a Slack alert.

How it works

  1. list_predictions (io.kestra.plugin.core.http.Request) calls GET /v1/predictions?created_after=... with a bearer token, which returns the most recent page of up to 100 predictions created since the start of the lookback window. It is a read: no model is run and nothing is canceled. Retries cover short network blips, and allowFailed plus allowFailure keep an API error from aborting the flow so it can be reported instead.
  2. summarize_predictions (io.kestra.plugin.core.debug.Return) first guards for a missing or non-200 response (UNREACHABLE). Otherwise one jq query keeps predictions whose status is final (succeeded, failed or aborted), counts canceled ones and ones still starting or processing separately, computes the share that failed or were aborted before they started, and groups failed predictions by model and by error. Fewer than min_predictions finished predictions is LOW_VOLUME; at or above critical_failure_rate is CRITICAL; at or above warning_failure_rate is WARNING; otherwise HEALTHY.
  3. route_by_failures (io.kestra.plugin.core.flow.Switch) branches on that classification: HEALTHY and LOW_VOLUME log quietly, WARNING posts a Slack note, CRITICAL and UNREACHABLE page Slack. A defaults fallback logs anything unexpected.
  4. log_audit_result (io.kestra.plugin.core.log.Log) writes one audit line per run with the HTTP status and the full summary.
  5. The flow-level errors block pages Slack if the flow itself fails, and the hourly_check trigger runs at 20 minutes past every hour, shipped disabled: true.

What you get

  • An hourly, read-only check of model inference with no custom scripting and no extra plugin to install.
  • A status output (HEALTHY / WARNING / CRITICAL / LOW_VOLUME / UNREACHABLE) other flows can gate on, for example to pin a known-good model version or fall back to another provider while failures are high.
  • Slack messages with the failure rate, the model that fails most (owner/name) and the most common error message, which usually points at the cause.
  • An audit line on every run, separate from Slack.

Who it's for

  • Product teams shipping image, video, audio or text features on Replicate models.
  • ML engineers who run their own models on Replicate and want to hear about a broken version before users do.
  • Teams that already run their AI pipelines and alerting in Kestra.

Why orchestrate this with Kestra

Replicate's dashboard lists predictions and webhooks report them one at a time, but neither tells you that one in five predictions failed in the last hour. Kestra turns a plain, read-only API call into a scheduled, typed classification with retries, a failure path and an execution history, and the status output lets your generation flows react automatically.

Prerequisites

  • A Replicate account that runs predictions through the API, and an API token from the account settings.
  • A Slack incoming webhook for the WARNING, CRITICAL, UNREACHABLE and flow-failure alerts.

Secrets

  • REPLICATE_API_TOKEN: a Replicate API token for the account (or organization) whose predictions you want to watch.
  • SLACK_WEBHOOK: Slack incoming webhook used for every alert branch and the flow-failure alert.

In Kestra OSS, secrets are supplied as environment variables prefixed SECRET_ and base64-encoded, for example SECRET_REPLICATE_API_TOKEN=$(echo -n 'r8_xxxxxxxx' | base64), and read back with {{ secret('REPLICATE_API_TOKEN') }}. Use the Enterprise secrets backend for encryption at rest and access control.

Quick start

  1. Run a few predictions on any public model, plus one with an invalid input to produce a failure.
  2. Add REPLICATE_API_TOKEN and SLACK_WEBHOOK as secrets.
  3. Run the flow manually with a low min_predictions and compare the counts, model and error in the logs with the predictions list in the Replicate dashboard.
  4. Enable hourly_check.

How to extend

  • Follow next when you run more than 100 predictions in the window, or shorten lookback_hours (the summary reports more_pages).
  • Filter the jq to one model and give it its own limits, for example for the model behind a paid feature.
  • Add a p95 of metrics.predict_time to catch slow models as well as failing ones.
  • Store the last status in a KV store and alert only when it changes.

Inputs

  • lookback_hours: type INT, default 6.
  • warning_failure_rate: type FLOAT, default 5.0 (percent).
  • critical_failure_rate: type FLOAT, default 20.0 (percent).
  • min_predictions: type INT, default 20.

Outputs

  • status: one of HEALTHY, WARNING, CRITICAL, LOW_VOLUME, UNREACHABLE.
  • Example path: {{ fromJson(outputs.summarize_predictions.value).top_model }} for the model that fails most.

Links

Pitfalls

  • Only the first page (100 predictions, newest first) is read; high-volume accounts need paging with next or a shorter window.
  • canceled predictions are counted but not treated as failures, because they are usually canceled on purpose by your own code. aborted ones (terminated before they started, for example by a deadline) are counted as failures.
  • Predictions created from the web playground are included too; filter on source == "api" in the jq if you only care about production calls.
  • This flow only reads; it never creates, cancels or deletes a prediction.
See How

New to Kestra?

Use blueprints to kickstart your first workflows.