Ollama GPU VRAM Saturation and Eviction Sentinel

id: ollama-vram-saturation-and-eviction-sentinel
namespace: company.ai

description: Monitor Ollama LLM serving nodes for GPU VRAM exhaustion, CPU RAM
  spilling bottlenecks, and automate idle model eviction with Slack alerts.

triggers:
  - id: periodic_ollama_audit
    type: io.kestra.plugin.core.trigger.Schedule
    description: Periodic health and memory audit of the Ollama runtime daemon.
      Shipped disabled by default.
    cron: "*/5 * * * *"
    disabled: true

  - id: on_demand_webhook
    type: io.kestra.plugin.core.trigger.Webhook
    description: Trigger an on-demand audit from Kubernetes HPA, Prometheus
      Alertmanager, or MLOps pipelines.
    key: ollama-vram-audit

inputs:
  - id: ollama_endpoint
    type: STRING
    displayName: Ollama Host Endpoint
    defaults: "http://localhost:11434"
    description: Base HTTP URL of the target Ollama instance.
  - id: total_gpu_vram_gb
    type: FLOAT
    displayName: Total Available Physical GPU VRAM (GB)
    defaults: 24.0
    description: Total dedicated physical GPU VRAM capacity available on the host
      serving node in gigabytes.
  - id: vram_warning_threshold_pct
    type: FLOAT
    displayName: VRAM Warning Threshold (%)
    defaults: 80.0
    description: Allocation percentage threshold triggering a warning notification.
  - id: vram_critical_threshold_pct
    type: FLOAT
    displayName: VRAM Critical Threshold (%)
    defaults: 92.0
    description: Allocation percentage threshold triggering critical saturation
      alerts and optional eviction.
  - id: auto_evict_idle_models
    type: BOOL
    displayName: Auto-Evict Active Models Under Critical Saturation
    defaults: false
    description: When true and saturation reaches CRITICAL, automatically unloads
      active models to reclaim GPU VRAM.
  - id: slack_webhook_url
    type: STRING
    displayName: Slack Incoming Webhook URL
    defaults: ""
    description: Optional webhook URL to send warning and critical incident notifications.

tasks:
  - id: check_ollama_liveness
    type: io.kestra.plugin.core.http.Request
    description: Probe Ollama daemon version to confirm runtime reachability and
      service liveness.
    uri: "{{ inputs.ollama_endpoint }}/api/version"
    method: GET
    timeout: PT10S

  - id: fetch_running_models
    type: io.kestra.plugin.core.http.Request
    description: Query the Ollama process table for currently loaded models and GPU
      layer allocations.
    uri: "{{ inputs.ollama_endpoint }}/api/ps"
    method: GET
    timeout: PT15S

  - id: evaluate_vram_and_spilling
    type: io.kestra.plugin.scripts.python.Script
    description: Analyze active model footprints, detect CPU RAM offloading
      bottlenecks, and assess saturation.
    containerImage: python:3.12-slim
    inputFiles:
      models_response.json: "{{ outputs.fetch_running_models.body }}"
    outputFiles:
      - metrics_summary.json
    script: |
      import json
      import sys

      with open("models_response.json", "r") as f:
          raw_data = json.load(f)

      models = raw_data.get("models", [])
      total_vram_bytes = {{ inputs.total_gpu_vram_gb }} * (1024 ** 3)
      warning_pct = {{ inputs.vram_warning_threshold_pct }}
      critical_pct = {{ inputs.vram_critical_threshold_pct }}

      total_allocated_vram = 0
      total_model_size = 0
      model_details = []
      spilled_models = []

      for m in models:
          m_name = m.get("name", "unknown")
          m_size = m.get("size", 0)
          m_vram = m.get("size_vram", 0)
          m_expires = m.get("expires_at", "never")

          total_allocated_vram += m_vram
          total_model_size += m_size

          # If size_vram is less than total size, layers were offloaded to system RAM
          is_spilled = m_vram < m_size
          if is_spilled:
              spilled_models.append(m_name)

          vram_gb = round(m_vram / (1024 ** 3), 2)
          total_gb = round(m_size / (1024 ** 3), 2)

          model_details.append({
              "name": m_name,
              "vram_gb": vram_gb,
              "total_gb": total_gb,
              "spilled_to_system_ram": is_spilled,
              "expires_at": m_expires
          })

      vram_usage_pct = round((total_allocated_vram / total_vram_bytes) * 100, 2) if total_vram_bytes > 0 else 0.0

      if vram_usage_pct >= critical_pct or len(spilled_models) > 0:
          status = "CRITICAL"
      elif vram_usage_pct >= warning_pct:
          status = "WARNING"
      else:
          status = "HEALTHY"

      metrics = {
          "status": status,
          "active_model_count": len(models),
          "allocated_vram_gb": round(total_allocated_vram / (1024 ** 3), 2),
          "total_vram_gb": {{ inputs.total_gpu_vram_gb }},
          "vram_utilization_pct": vram_usage_pct,
          "spilled_models_count": len(spilled_models),
          "spilled_models": spilled_models,
          "models": model_details
      }

      print(f"[Ollama Sentinel] Health: {status} | VRAM: {vram_usage_pct}% | Active Models: {len(models)}")
      if spilled_models:
          print(f"[Warning] Models spilled to CPU RAM (severe inference latency risk): {spilled_models}")

      with open("metrics_summary.json", "w") as out:
          json.dump(metrics, out, indent=2)

      print("::" + json.dumps({"outputs": metrics}) + "::")

  - id: check_and_remediate_idle_models
    type: io.kestra.plugin.core.flow.If
    description: Proactively unload models with keep_alive:0 if critical VRAM
      pressure is detected and auto_evict is enabled.
    condition: "{{ inputs.auto_evict_idle_models and
      outputs.evaluate_vram_and_spilling.vars.status == 'CRITICAL' and
      outputs.evaluate_vram_and_spilling.vars.active_model_count > 0 }}"
    then:
      - id: log_remediation_intent
        type: io.kestra.plugin.core.log.Log
        description: Log intention to evict models under memory saturation.
        message: "Triggering proactive eviction of active Ollama models to relieve
          critical VRAM saturation: {{
          outputs.evaluate_vram_and_spilling.vars.models }}"

      - id: evict_each_model
        type: io.kestra.plugin.core.flow.Loop
        values: "{{ outputs.evaluate_vram_and_spilling.vars.models }}"
        tasks:
          - id: unload_model
            type: io.kestra.plugin.core.http.Request
            description: Unload model from GPU memory immediately by passing keep_alive = 0.
            uri: "{{ inputs.ollama_endpoint }}/api/generate"
            method: POST
            contentType: application/json
            body: |
              {
                "model": "{{ item.value.name }}",
                "keep_alive": 0
              }

  - id: dispatch_slack_alert
    type: io.kestra.plugin.core.flow.If
    description: Notify engineering team on Slack if memory headroom is in WARNING
      or CRITICAL state.
    condition: "{{ inputs.slack_webhook_url != null and inputs.slack_webhook_url !=
      '' and outputs.evaluate_vram_and_spilling.vars.status != 'HEALTHY' }}"
    then:
      - id: post_slack_notification
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        description: Dispatch structured Slack notification with memory utilization and
          spilling details.
        url: "{{ inputs.slack_webhook_url }}"
        payload: |
          {
            "text": ":rotating_light: *Ollama Sentinel Alert: {{ outputs.evaluate_vram_and_spilling.vars.status }}*\n*Host Endpoint:* `{{ inputs.ollama_endpoint }}`\n*VRAM Utilization:* `{{ outputs.evaluate_vram_and_spilling.vars.vram_utilization_pct }}%` ({{ outputs.evaluate_vram_and_spilling.vars.allocated_vram_gb }} GB / {{ inputs.total_gpu_vram_gb }} GB)\n*Active Models:* {{ outputs.evaluate_vram_and_spilling.vars.active_model_count }}\n*CPU Spilled Models:* `{{ outputs.evaluate_vram_and_spilling.vars.spilled_models }}`\n*Auto Eviction Run:* {{ inputs.auto_evict_idle_models }}\n*Execution:* `{{ execution.id }}`"
          }

errors:
  - id: alert_audit_failure
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    description: Alert on Slack when Ollama daemon is unreachable or returns HTTP errors.
    url: "{{ inputs.slack_webhook_url }}"
    payload: |
      {
        "text": ":warning: *Ollama Telemetry Sentinel Failed*\n*Endpoint:* {{ inputs.ollama_endpoint }}\n*Execution:* {{ execution.id }}\nCheck Ollama daemon process liveness, host networking, or firewall settings."
      }

outputs:
  - id: metrics_summary
    type: JSON
    description: Aggregated telemetry output including VRAM utilization, RAM
      spilling, and operational status.
    value: "{{ outputs.evaluate_vram_and_spilling.vars | toJson }}"

Running local large language models (LLMs) with Ollama on shared GPU servers or workstations frequently leads to silent memory saturation. When new inference tasks load additional models, existing models remain resident in GPU VRAM until their keep_alive window expires. If cumulative weights exceed physical VRAM capacity, Ollama either fails with CUDA out-of-memory (OOM) errors or silently spills model layers onto system RAM, resulting in orders-of-magnitude slower token generation.

This blueprint provides an autonomous reliability sentinel that periodically inspects Ollama's active model telemetry, calculates physical GPU VRAM saturation ratios, identifies models operating in hybrid CPU/RAM offload mode, and optionally triggers proactive model unloads (keep_alive: 0) to safeguard cluster headroom.

How it works

check_ollama_liveness validates that the Ollama HTTP daemon is responsive by polling the /api/version endpoint.
fetch_running_models extracts model metadata, total size, VRAM footprint (size_vram), and expiration timestamps from /api/ps.
evaluate_vram_and_spilling executes an in-container Python analysis task that compares active allocations against configured physical GPU memory (total_gpu_vram_gb), detects any layer spilling (size_vram < size), and assigns operational health tiers (HEALTHY, WARNING, CRITICAL).
check_and_remediate_idle_models evaluates whether critical saturation was reached and, if auto_evict_idle_models is enabled, iterates through resident models using a Loop flowable task to trigger immediate GPU cache release via /api/generate with keep_alive: 0.
dispatch_slack_alert posts an actionable diagnostic card to Slack whenever thresholds are breached or CPU RAM spilling is detected.
The errors block catches connection or runtime timeouts and notifies Slack.

What you get

Automated GPU memory headroom audit without running invasive host monitoring agents.
Early detection of CPU RAM layer offloading, which causes severe inference throughput collapse.
Configurable threshold-based alerting (WARNING and CRITICAL) with detailed utilization statistics.
Automated self-healing eviction workflow to reclaim VRAM without restarting the Ollama daemon.

Who it's for

Platform engineers and MLOps teams hosting shared Ollama instances for developers or internal applications.
AI engineers serving multiple fine-tuned or multi-size models (e.g., Llama 3, Mistral, Qwen) on limited GPU hardware.
DevOps engineers looking to automate GPU cluster hygiene and eliminate manual process restarts.

Why orchestrate this with Kestra

Declarative workflows eliminate the need for custom Python cron daemons or fragile bash scripts.
Built-in conditional execution (If) and iteration (Loop) manage multi-step remediation safely.
Native alerting tasks (SlackIncomingWebhook) dispatch actionable diagnostic payloads directly into team channels without third-party integrations.
Centralized execution history provides auditable logs of all model evictions and saturation events.

Prerequisites

A running Ollama instance accessible over HTTP (default: http://localhost:11434).
Python runtime container execution capability on the Kestra worker (Docker or Process runner).
An optional Slack Incoming Webhook URL for alerting.

Secrets

SLACK_WEBHOOK_URL: Optional Slack Incoming Webhook URL configured in namespace secrets or passed via inputs.

Quick start

Adjust ollama_endpoint to match your Ollama host (e.g., http://ollama-service.mlops.svc.cluster.local:11434).
Set total_gpu_vram_gb to the physical VRAM size of your GPU (e.g., 24.0 for RTX 3090/4090/A5000, 80.0 for A100/H100).
Configure slack_webhook_url if you want team channel alerts.
Enable auto_evict_idle_models: true if you want Kestra to automatically unload models when memory exceeds vram_critical_threshold_pct.
Enable the schedule trigger or call the webhook trigger ollama-vram-audit from your CI/CD or monitoring system.

How to extend

Add Prometheus pushgateway tasks to export VRAM metrics into your Grafana dashboards.
Pair with an Ollama model preload task that warms specific high-priority models during off-peak hours.
Implement priority-based eviction that only unloads models matching specific name patterns or tags.

Ollama GPU VRAM Saturation and Idle Model Eviction Sentinel

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Ollama GPU VRAM Saturation and Idle Model Eviction Sentinel

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More AI Blueprints

Langfuse Daily LLM Trace Quality and Cost Sentinel

LLM Evaluation & Regression Testing

Watch LLM Model Prices and Releases with an AI Briefing in Slack

LLM Model Retirement Migrator for OpenAI, Anthropic and Gemini Model IDs in Your Repositories

LLM Token and Dollar Budget Guardrail for Teams

LiteLLM Proxy Budget Quota & RPM Fallback Circuit Breaker

More Infrastructure Blueprints

KMS Key Rotation Compliance Gate

KeyDB Active-Replica Multi-Master Peering Sentinel

Backport Kestra UI hotfixes to a Git review branch

Gate Kestra Subflow Changes Against Every Caller Before Deploying

Audit Keycloak Realm Security and Fix It with On-Call Approval

Knative Service Revision Failure and Scale Sentinel

New to Kestra?