Schedule icon
WorkingDirectory icon
Script icon
Process icon
If icon
Loop icon
StructuredOutputCompletion icon
Parallel icon
Create icon
SlackIncomingWebhook icon

Regression-Test Your AI Agent Nightly with a Golden-Prompt Harness

Nightly golden-prompt regression harness for AI agents, with deterministic scoring, per-category pass rates, trend tracking, Gemini-judged failures, GitHub issues, and Slack alerts.

Categories
AI

What it does

Agents regress quietly. A model upgrade, a system-prompt tweak, or a new tool can break instruction following, tool selection, or refusal behaviour that worked last week, and nobody notices until a user does. This blueprint turns the agent's expected behaviour into a golden dataset and replays it on a schedule: every reply is scored deterministically against the golden expectation, the run produces a per-category pass-rate scorecard with a trend against the previous run, and failing categories are judged by Gemini (a real regression, or a golden set that went stale) and escalated to a GitHub issue plus a Slack alert. It is the functional-correctness companion to the red-team probe harness: that one tests whether the agent resists attacks, this one tests whether it still does its job.

Prerequisites

An agent endpoint that accepts a POST with a prompt and returns a reply as JSON (the agent_endpoint input, with response_json_path pointing at the reply text). Three namespace secrets: GEMINI_API_KEY for the judging stage, GITHUB_TOKEN with issue-write scope on the tracking repository, and SLACK_WEBHOOK_URL for the alert channel. A golden dataset is optional at first: the bundled 12-case starter set (3 cases in each of instruction following, tool selection, refusal behaviour, and factual QA) runs with no dataset connected.

How it works

  1. nightly_regression (io.kestra.plugin.core.trigger.Schedule) runs the harness nightly at 06:00 UTC. Run it on demand after any model or system-prompt change.
  2. load_golden_dataset (Python scripts) loads the golden dataset from a URL or writes the bundled 12-case starter set: instruction following, tool selection, refusal behaviour, and factual QA.
  3. run_cases (Python) replays every case against the agent endpoint and captures the raw replies, recording transport errors per case rather than aborting the run.
  4. score_cases (Python) checks required and forbidden text, accepted phrases, and the tool the agent chose, then builds the scorecard artifact (scorecard.json and scorecard.md) with per-category pass rates and a trend against the previous scorecard.
  5. triage (Kestra flow tasks) branches on the gate: passing runs are recorded; failing runs go to judgement and escalation.
  6. judge_weak_categories (Gemini) decides for each weak category whether the agent regressed or the golden set went stale.
  7. escalate opens a GitHub issue and posts a Slack alert in parallel, while write_trend_record appends the run to the trend series.

Quick start

  1. Add the three secrets above to the namespace.
  2. Run the flow once with no inputs: the starter set exercises the full pipeline and produces a scorecard.
  3. Read scorecard.md for the per-category rates and the gate verdict.
  4. Point agent_endpoint at your agent and golden_dataset_url at your own cases, then point previous_scorecard_url at each published scorecard so the trend builds.

Expected outputs

scorecard.json and scorecard.md (overall and per-category pass rates, the gate verdict, and the trend against the previous run), a trend.jsonl record per run, and on a failed gate one GitHub issue plus one Slack alert naming the weak categories and the judge's verdict per category.

How to adapt it

Replace the bundled 12-case starter dataset with golden cases from your own product (golden_dataset_url), grouped by the categories you care about. Tune min_pass_rate and min_category_rate to your release bar, publish each run's scorecard and point previous_scorecard_url at it so the trend builds, and swap the Gemini judge model in judge_weak_categories if your organisation standardises on another provider.

See How

New to Kestra?

Use blueprints to kickstart your first workflows.