Promptfoo LLM Red-Teaming and Security Regression Gate
Run automated LLM red-teaming, prompt injection defense, and security regression gates using Promptfoo before deploying AI models.
Read recent Replicate predictions every hour, run no model, and alert Slack when failures cross your limits, with the failing model and top error.
Image generation, transcription, upscaling and LLM features built on Replicate fail one prediction at a time: a model owner pushes a new version, an input field changes type, a GPU runs out of memory on large inputs, or the account hits a spend limit. Each prediction just ends with status: failed and an error string, the app shows a generic error, and nobody notices the rate climbing until users complain. This blueprint turns a scheduled, read-only call to the Replicate API into an inference health gate: io.kestra.plugin.core.http.Request reads recent predictions, one jq query computes the failure rate, the model that fails most and the most common error, and the result routes to a quiet log or a Slack alert.
list_predictions (io.kestra.plugin.core.http.Request) calls GET /v1/predictions?created_after=... with a bearer token, which returns the most recent page of up to 100 predictions created since the start of the lookback window. It is a read: no model is run and nothing is canceled. Retries cover short network blips, and allowFailed plus allowFailure keep an API error from aborting the flow so it can be reported instead.summarize_predictions (io.kestra.plugin.core.debug.Return) first guards for a missing or non-200 response (UNREACHABLE). Otherwise one jq query keeps predictions whose status is final (succeeded, failed or aborted), counts canceled ones and ones still starting or processing separately, computes the share that failed or were aborted before they started, and groups failed predictions by model and by error. Fewer than min_predictions finished predictions is LOW_VOLUME; at or above critical_failure_rate is CRITICAL; at or above warning_failure_rate is WARNING; otherwise HEALTHY.route_by_failures (io.kestra.plugin.core.flow.Switch) branches on that classification: HEALTHY and LOW_VOLUME log quietly, WARNING posts a Slack note, CRITICAL and UNREACHABLE page Slack. A defaults fallback logs anything unexpected.log_audit_result (io.kestra.plugin.core.log.Log) writes one audit line per run with the HTTP status and the full summary.errors block pages Slack if the flow itself fails, and the hourly_check trigger runs at 20 minutes past every hour, shipped disabled: true.status output (HEALTHY / WARNING / CRITICAL / LOW_VOLUME / UNREACHABLE) other flows can gate on, for example to pin a known-good model version or fall back to another provider while failures are high.owner/name) and the most common error message, which usually points at the cause.Replicate's dashboard lists predictions and webhooks report them one at a time, but neither tells you that one in five predictions failed in the last hour. Kestra turns a plain, read-only API call into a scheduled, typed classification with retries, a failure path and an execution history, and the status output lets your generation flows react automatically.
REPLICATE_API_TOKEN: a Replicate API token for the account (or organization) whose predictions you want to watch.SLACK_WEBHOOK: Slack incoming webhook used for every alert branch and the flow-failure alert.In Kestra OSS, secrets are supplied as environment variables prefixed SECRET_ and base64-encoded, for example SECRET_REPLICATE_API_TOKEN=$(echo -n 'r8_xxxxxxxx' | base64), and read back with {{ secret('REPLICATE_API_TOKEN') }}. Use the Enterprise secrets backend for encryption at rest and access control.
REPLICATE_API_TOKEN and SLACK_WEBHOOK as secrets.min_predictions and compare the counts, model and error in the logs with the predictions list in the Replicate dashboard.hourly_check.next when you run more than 100 predictions in the window, or shorten lookback_hours (the summary reports more_pages).model and give it its own limits, for example for the model behind a paid feature.metrics.predict_time to catch slow models as well as failing ones.lookback_hours: type INT, default 6.warning_failure_rate: type FLOAT, default 5.0 (percent).critical_failure_rate: type FLOAT, default 20.0 (percent).min_predictions: type INT, default 20.status: one of HEALTHY, WARNING, CRITICAL, LOW_VOLUME, UNREACHABLE.{{ fromJson(outputs.summarize_predictions.value).top_model }} for the model that fails most.next or a shorter window.canceled predictions are counted but not treated as failures, because they are usually canceled on purpose by your own code. aborted ones (terminated before they started, for example by a deadline) are counted as failures.source == "api" in the jq if you only care about production calls.