Compare Local Ollama Models Head-to-Head

id: ai-ollama-model-bakeoff
namespace: company.team

description: |
  Head-to-head bakeoff for local Ollama models. The flow sends the same prompt
  to every model on your list, one at a time, then renders a side-by-side
  leaderboard so you can pick the best model for your workload before
  committing it to production.

inputs:
  - id: prompt
    type: STRING
    displayName: Evaluation prompt
    description: The prompt every model answers. Pick something representative of
      your real workload.
    defaults: "Explain what a workflow orchestrator does in two sentences, for a
      junior developer."
  - id: models
    type: ARRAY
    itemType: STRING
    displayName: Models to compare
    description: Ollama model names to bake off. They must already be pulled. Runs
      sequentially so latency numbers are fair.
    defaults:
      - llama3.2:3b
      - qwen2.5:7b
      - mistral:7b
  - id: ollama_endpoint
    type: STRING
    displayName: Ollama endpoint
    description: Base URL where Ollama listens. Use
      http://host.docker.internal:11434 when Kestra runs in Docker and Ollama
      runs on the host.
    defaults: "http://localhost:11434"

tasks:
  - id: bakeoff
    type: io.kestra.plugin.core.flow.Loop
    description: Ask each model the same prompt. Sequential (concurrencyLimit 1) so
      latency is comparable across models on the same hardware.
    values: "{{ inputs.models }}"
    concurrencyLimit: 1
    tasks:
      - id: ask
        type: io.kestra.plugin.ai.completion.ChatCompletion
        description: Send the evaluation prompt to the current model via the Ollama
          provider.
        provider:
          type: io.kestra.plugin.ai.provider.Ollama
          modelName: "{{ item.value }}"
          endpoint: "{{ inputs.ollama_endpoint }}"
        messages:
          - type: SYSTEM
            content: You are a helpful assistant. Answer concisely.
          - type: USER
            content: "{{ inputs.prompt }}"
    outputs:
      - id: model
        type: STRING
        description: The model that produced this row of the leaderboard.
        value: "{{ item.value }}"
      - id: answer
        type: STRING
        description: The model's raw answer to the evaluation prompt.
        value: "{{ outputs.ask.textOutput ?? '(no text returned)' }}"

  - id: leaderboard
    type: io.kestra.plugin.core.log.Log
    description: Print the side-by-side comparison so the winner is obvious at a glance.
    message: |
      ## Ollama bakeoff results ({{ inputs.models | length }} models)
      Prompt: "{{ inputs.prompt }}"
      {% for row in outputs.bakeoff %}
      ### {{ row.model }}
      {{ row.answer }}

      {% endfor %}

  - id: save_report
    type: io.kestra.plugin.core.storage.Write
    description: Persist the leaderboard as a markdown report in Kestra internal
      storage for sharing and record-keeping.
    content: |
      # Ollama model bakeoff — {{ execution.startDate | date("yyyy-MM-dd HH:mm") }}

      **Prompt:** {{ inputs.prompt }}

      {% for row in outputs.bakeoff %}
      ## `{{ row.model }}`

      {{ row.answer }}

      ---
      {% endfor %}

      *Generated by the `ai-ollama-model-bakeoff` blueprint (execution {{ execution.id }}).*
    extension: .md