LLM Evaluation & Regression Testing with Kestra

id: llm-evaluation-regression

namespace: hacktober.ai

description: |
  Evaluate LLM responses against a golden dataset using LLM-as-a-Judge,
  calculate quality metrics, and detect quality regressions.

triggers:
  - id: daily_evaluation
    type: io.kestra.plugin.core.trigger.Schedule
    description: Run the LLM evaluation daily at 09:00 IST.
    cron: "0 9 * * *"
    timezone: Asia/Kolkata
    disabled: true

inputs:

  - id: model_name
    type: STRING
    defaults: openai/gpt-oss-120b
    description: LLM model used for response generation and evaluation.

  - id: quality_threshold
    type: FLOAT
    defaults: 0.85
    description: Minimum acceptable overall evaluation score.

  - id: baseline_score
    type: FLOAT
    defaults: 0.90
    description: Baseline score used for regression detection.

  - id: regression_tolerance
    type: FLOAT
    defaults: 0.05
    description: Allowed decrease from the baseline before regression is reported.

  - id: golden_dataset
    type: JSON
    defaults:
      - id: qa_001
        question: What is the capital of France?
        expected: Paris
        context: Paris is the capital and largest city of France.

      - id: qa_002
        question: What is 15 multiplied by 4?
        expected: "60"
        context: The mathematical result of 15 multiplied by 4 is 60.

      - id: qa_003
        question: What programming language is primarily used to build Android
          applications?
        expected: Kotlin
        context: Kotlin is a primary programming language used for Android application
          development.

    description: |
      Golden evaluation dataset.

      Each item must contain:
      - id
      - question
      - expected
      - context

tasks:

  - id: evaluate_dataset
    type: io.kestra.plugin.core.flow.Loop
    description: Evaluate every item in the golden dataset.
    values: "{{ inputs.golden_dataset }}"
    concurrencyLimit: 3
    fetchType: FETCH

    tasks:

      - id: generate_answer
        type: io.kestra.plugin.ai.completion.ChatCompletion
        description: Generate an answer from the configured LLM.

        provider:
          type: io.kestra.plugin.ai.provider.OpenAI
          apiKey: "{{ secret('GROQ_API_KEY') }}"
          baseUrl: https://api.groq.com/openai/v1
          modelName: "{{ inputs.model_name }}"

        messages:

          - type: SYSTEM
            content: |
              You are a helpful assistant.
              Answer the user's question accurately and concisely.

          - type: USER
            content: "{{ fromJson(item.value).question }}"

      - id: judge
        type: io.kestra.plugin.ai.completion.ChatCompletion
        description: Evaluate the generated answer using LLM-as-a-Judge.

        provider:
          type: io.kestra.plugin.ai.provider.OpenAI
          apiKey: "{{ secret('GROQ_API_KEY') }}"
          baseUrl: https://api.groq.com/openai/v1
          modelName: "{{ inputs.model_name }}"

        messages:

          - type: SYSTEM
            content: |
              You are an expert LLM evaluation judge.

              Evaluate the model answer against the expected answer
              and provided context.

              Score each dimension from 0.0 to 1.0.

              correctness:
              Measures whether the answer is factually correct
              compared with the expected answer.

              relevance:
              Measures whether the answer directly addresses
              the question without unnecessary information.

              groundedness:
              Measures whether the answer is supported by the
              provided context.

              The aggregate_group field must always be "all".

              Return the test metadata exactly as provided.

              Return ONLY valid JSON matching the provided schema.

          - type: USER
            content: |
              Test ID:
              {{ fromJson(item.value).id }}

              Question:
              {{ fromJson(item.value).question }}

              Expected Answer:
              {{ fromJson(item.value).expected }}

              Context:
              {{ fromJson(item.value).context }}

              Model Answer:
              {{ outputs.generate_answer.textOutput }}

        configuration:
          responseFormat:
            type: JSON
            jsonSchema:
              type: object

              properties:

                test_id:
                  type: string

                question:
                  type: string

                expected_answer:
                  type: string

                model_answer:
                  type: string

                aggregate_group:
                  type: string

                correctness:
                  type: number

                relevance:
                  type: number

                groundedness:
                  type: number

                reason:
                  type: string

              required:
                - test_id
                - question
                - expected_answer
                - model_answer
                - aggregate_group
                - correctness
                - relevance
                - groundedness
                - reason

    outputs:

      - id: evaluation
        type: JSON
        value: "{{ outputs.judge.jsonOutput }}"

  - id: aggregate_scores
    type: io.kestra.plugin.transform.Aggregate
    description: Aggregate evaluation scores across the golden dataset.

    from: "{{ loopOutputs(outputs.evaluate_dataset.outputs, 'evaluation') }}"

    groupBy:
      - aggregate_group

    aggregates:

      average_correctness:
        expr: avg(correctness)
        type: DECIMAL

      average_relevance:
        expr: avg(relevance)
        type: DECIMAL

      average_groundedness:
        expr: avg(groundedness)
        type: DECIMAL

      test_count:
        expr: count()
        type: INT

    outputType: RECORDS

  - id: calculate_quality
    type: io.kestra.plugin.core.output.OutputValues
    description: Calculate the overall LLM quality score.

    values:

      average_correctness: "{{ outputs.aggregate_scores.records[0].average_correctness }}"

      average_relevance: "{{ outputs.aggregate_scores.records[0].average_relevance }}"

      average_groundedness: "{{ outputs.aggregate_scores.records[0].average_groundedness }}"

      test_count: "{{ outputs.aggregate_scores.records[0].test_count }}"

      overall_score: >-
        {{
          (
            outputs.aggregate_scores.records[0].average_correctness
            +
            outputs.aggregate_scores.records[0].average_relevance
            +
            outputs.aggregate_scores.records[0].average_groundedness
          ) / 3
        }}

  - id: evaluation_report
    type: io.kestra.plugin.core.log.Log
    description: Log the aggregate evaluation report.

    message: |
      ==========================================
      LLM EVALUATION REPORT
      ==========================================

      Model:
      {{ inputs.model_name }}

      Tests:
      {{ outputs.calculate_quality.values.test_count }}

      Average Correctness:
      {{ outputs.calculate_quality.values.average_correctness }}

      Average Relevance:
      {{ outputs.calculate_quality.values.average_relevance }}

      Average Groundedness:
      {{ outputs.calculate_quality.values.average_groundedness }}

      Overall Score:
      {{ outputs.calculate_quality.values.overall_score }}

      Quality Threshold:
      {{ inputs.quality_threshold }}

      Baseline Score:
      {{ inputs.baseline_score }}

      Regression Tolerance:
      {{ inputs.regression_tolerance }}

      ==========================================

  - id: individual_test_results
    type: io.kestra.plugin.core.log.Log
    description: Log the evaluation result for every test case.

    message: |
      ==========================================
      INDIVIDUAL TEST RESULTS
      ==========================================

      {{ loopOutputs(outputs.evaluate_dataset.outputs, 'evaluation') }}

      ==========================================

  - id: evaluation_results
    type: io.kestra.plugin.core.output.OutputValues
    description: Expose evaluation metrics as reusable flow outputs.

    values:

      model_name: "{{ inputs.model_name }}"

      test_count: "{{ outputs.calculate_quality.values.test_count }}"

      average_correctness: "{{ outputs.calculate_quality.values.average_correctness }}"

      average_relevance: "{{ outputs.calculate_quality.values.average_relevance }}"

      average_groundedness: "{{ outputs.calculate_quality.values.average_groundedness }}"

      overall_score: "{{ outputs.calculate_quality.values.overall_score }}"

      quality_threshold: "{{ inputs.quality_threshold }}"

      baseline_score: "{{ inputs.baseline_score }}"

      regression_tolerance: "{{ inputs.regression_tolerance }}"

  - id: quality_threshold_failure
    type: io.kestra.plugin.core.execution.Fail
    description: Fail when the overall quality score is below the required threshold.

    condition: >-
      {{
        (
          outputs.calculate_quality.values.overall_score
          | string
          | number('BIGDECIMAL')
        )
        <
        (
          inputs.quality_threshold
          | string
          | number('BIGDECIMAL')
        )
      }}

    errorMessage: |
      LLM QUALITY THRESHOLD FAILED.

      Overall Score:
      {{ outputs.calculate_quality.values.overall_score }}

      Required Threshold:
      {{ inputs.quality_threshold }}

  - id: regression_failure
    type: io.kestra.plugin.core.execution.Fail
    description: Fail when the quality score falls below the regression boundary.

    condition: >-
      {{
        (
          outputs.calculate_quality.values.overall_score
          | string
          | number('BIGDECIMAL')
        )
        <
        (
          (
            inputs.baseline_score
            | string
            | number('BIGDECIMAL')
          )
          -
          (
            inputs.regression_tolerance
            | string
            | number('BIGDECIMAL')
          )
        )
      }}

    errorMessage: |
      LLM REGRESSION DETECTED.

      Overall Score:
      {{ outputs.calculate_quality.values.overall_score }}

      Baseline Score:
      {{ inputs.baseline_score }}

      Regression Tolerance:
      {{ inputs.regression_tolerance }}

      Regression Boundary:
      {{
        inputs.baseline_score - inputs.regression_tolerance
      }}

  - id: evaluation_passed
    type: io.kestra.plugin.core.log.Log
    description: Confirm that all quality and regression checks passed.

    message: |
      ==========================================
      LLM EVALUATION PASSED
      ==========================================

      Model:
      {{ inputs.model_name }}

      Overall Score:
      {{ outputs.calculate_quality.values.overall_score }}

      Quality Threshold:
      {{ inputs.quality_threshold }}

      Baseline:
      {{ inputs.baseline_score }}

      Regression Boundary:
      {{
        inputs.baseline_score - inputs.regression_tolerance
      }}

      All quality and regression checks passed.

      ==========================================

  - id: slack_success_notification
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    description: Send an optional Slack notification when evaluation passes.
    url: "{{ secret('SLACK_WEBHOOK') }}"
    messageText: |
      *LLM Evaluation Passed* :white_check_mark:

      Model: {{ inputs.model_name }}

      Overall Score:
      {{ outputs.calculate_quality.values.overall_score }}

      Quality Threshold:
      {{ inputs.quality_threshold }}

      Baseline:
      {{ inputs.baseline_score }}

      Regression Tolerance:
      {{ inputs.regression_tolerance }}

      Tests:
      {{ outputs.calculate_quality.values.test_count }}
    disabled: true

errors:

  - id: slack_failure_notification
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    description: Send an optional Slack notification when evaluation fails.
    url: "{{ secret('SLACK_WEBHOOK') }}"
    messageText: |
      *LLM Evaluation Failed* :x:

      Model: {{ inputs.model_name }}

      Overall Score:
      {{ outputs.calculate_quality.values.overall_score }}

      Quality Threshold:
      {{ inputs.quality_threshold }}

      Baseline:
      {{ inputs.baseline_score }}

      Regression Tolerance:
      {{ inputs.regression_tolerance }}
    disabled: true

Build a reusable evaluation pipeline for Large Language Model applications. This blueprint runs a model against a configurable golden dataset, evaluates every response using LLM-as-a-Judge, aggregates quality metrics, and detects regressions against a configurable baseline.

How it works

The daily_evaluation schedule trigger is disabled by default and can be enabled for recurring evaluations.
The evaluate_dataset loop processes every item in the golden dataset with configurable concurrency.
The generate_answer task sends each question to the configured LLM through the OpenAI-compatible Groq endpoint.
The judge task evaluates each response for correctness, relevance, and groundedness using structured JSON output.
The aggregate_scores task calculates average scores across the complete evaluation dataset.
The calculate_quality task calculates a single overall quality score from the three evaluation dimensions.
The quality_threshold_failure task fails the execution when the overall score is below the required quality threshold.
The regression_failure task detects quality degradation relative to the configured baseline and tolerance.
Optional Slack notifications can be enabled for successful and failed evaluations.

What you get

Per-test evaluation results.
Correctness, relevance, and groundedness metrics.
A single overall quality score.
Configurable minimum quality threshold.
Configurable regression baseline and tolerance.
Parallelized dataset evaluation through Kestra's Loop task.
Structured JSON evaluation results.
Optional Slack notifications.
A reusable evaluation workflow that can be adapted to different LLM applications.

Who it's for

AI and ML engineers testing LLM applications.
Teams maintaining production RAG or GenAI applications.
Developers introducing automated LLM regression testing.
Platform teams building repeatable model evaluation pipelines.

Why orchestrate LLM evaluation with Kestra

LLM applications can change behavior when models, prompts, providers, or application code change. Manual testing does not scale well for repeated evaluations.

Kestra provides declarative orchestration, concurrency, scheduling, structured outputs, execution history, failure handling, and reusable workflow configuration in a single YAML definition.

Prerequisites

A Kestra instance.
An OpenAI-compatible LLM provider.
A Groq API key for the default configuration.
A GROQ_API_KEY Kestra secret.
Optional SLACK_WEBHOOK secret if Slack notifications are enabled.

Quick start

Configure the GROQ_API_KEY secret in your Kestra namespace.
Review the golden_dataset input and replace the examples with your own evaluation cases.
Adjust model_name, quality_threshold, baseline_score, and regression_tolerance as required.
Execute the flow manually to inspect individual and aggregate evaluation results.
Enable the daily_evaluation trigger when recurring evaluation is required.
Configure SLACK_WEBHOOK and remove disabled: true from the Slack notification task and error handler if Slack alerts are desired.

How to extend

Replace the default OpenAI-compatible provider with another supported Kestra AI provider.
Add additional evaluation dimensions such as factuality, safety, or task success.
Add deterministic code-based evaluators for metrics that do not require an LLM judge.
Store historical evaluation scores in a database for long-term regression tracking.
Add model comparison by evaluating multiple model inputs against the same golden dataset.
Add downstream reporting or dashboard tasks using the generated evaluation outputs.

LLM Evaluation & Regression Testing

How it works

What you get

Who it's for

Why orchestrate LLM evaluation with Kestra

Prerequisites

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

LLM Evaluation & Regression Testing

How it works

What you get

Who it's for

Why orchestrate LLM evaluation with Kestra

Prerequisites

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More AI Blueprints

Decide When to Take Action Using AI, Check Real-Time Weather and Trigger Automations

AI Customer Call Compliance and Regulatory Disclosure Auditor

Automate Customer Feedback Triage, Sentiment Analysis, and Escalation with Google Gemini and Slack

Convert Natural Language Prompts to Structured Todoist Tasks with Deepseek AI

AI Contract Clause Risk Analyzer and Legal Sign-Off Gate

AI Annual Report Finder and Data Strategy Summary

New to Kestra?