LLM Gateway with vLLM, Gemini Fallback and Langfuse in Kestra

id: llm-gateway-fallback-vllm-to-gemini-with-langfuse
namespace: company.ai

description: |
  Production LLM Gateway routing prompts to a cost-effective local vLLM or Ollama instance,
  with automatic resilience failover to Google Gemini on error or timeout, and centralized
  latency, cost, and routing observability tracked in Langfuse.

inputs:
  - id: prompt
    type: STRING
    displayName: User Prompt
    description: Prompt payload to route through the LLM gateway.
    defaults: "Summarize the key architectural principles of event-driven workflow
      orchestration in three concise bullet points."

  - id: system_prompt
    type: STRING
    displayName: System Prompt
    description: System instructions guiding the tone and constraints of the assistant.
    defaults: "You are an expert enterprise software architect. Provide direct,
      concise, and structured answers."

  - id: vllm_endpoint
    type: STRING
    displayName: Local vLLM / Ollama Base URL
    description: Host endpoint for local inference (use
      http://host.docker.internal:11434 when Kestra runs inside Docker).
    defaults: "http://localhost:11434"

  - id: vllm_model
    type: STRING
    displayName: Local Model Name
    description: Name of the local model hosted on vLLM or Ollama (must be pulled in
      advance).
    defaults: "llama3.2:3b"

  - id: gemini_model
    type: STRING
    displayName: Cloud Fallback Model
    description: Google Gemini model version to serve as the high-availability failover.
    defaults: "gemini-3.1-flash-lite"

triggers:
  - id: on_request
    type: io.kestra.plugin.core.trigger.Webhook
    description: Authenticated HTTP webhook endpoint enabling upstream microservices
      or apps to query the gateway.
    key: "{{ secret('WEBHOOK_KEY') }}"

  - id: nightly_health_check
    type: io.kestra.plugin.core.trigger.Schedule
    description: Automated sanity check validating endpoint availability and
      fallback resilience. Shipped disabled by default.
    cron: "0 8 * * 1-5"
    disabled: true

tasks:
  - id: call_local_vllm
    type: io.kestra.plugin.ai.completion.ChatCompletion
    description: Query the local vLLM or Ollama model first for fast, zero-egress,
      zero-token-cost inference.
    allowFailure: true
    timeout: PT15S
    provider:
      type: io.kestra.plugin.ai.provider.Ollama
      endpoint: "{{ inputs.vllm_endpoint }}"
      modelName: "{{ inputs.vllm_model }}"
    messages:
      - type: SYSTEM
        content: "{{ inputs.system_prompt }}"
      - type: USER
        content: "{{ inputs.prompt }}"

  - id: check_fallback
    type: io.kestra.plugin.core.flow.If
    description: Evaluate if local inference timed out, failed, or was unreachable,
      and route to Google Gemini.
    condition: "{{ outputs.call_local_vllm.textOutput is not defined }}"
    then:
      - id: call_gemini_fallback
        type: io.kestra.plugin.ai.completion.ChatCompletion
        description: Cloud fallback querying Google Gemini to maintain uptime during
          local server outages.
        provider:
          type: io.kestra.plugin.ai.provider.GoogleGemini
          modelName: "{{ inputs.gemini_model }}"
          apiKey: "{{ secret('GEMINI_API_KEY') }}"
        messages:
          - type: SYSTEM
            content: "{{ inputs.system_prompt }}"
          - type: USER
            content: "{{ inputs.prompt }}"

      - id: notify_slack_fallback
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        description: Alert AI engineers on Slack when the gateway triggers a cloud failover.
        url: "{{ secret('SLACK_WEBHOOK_URL') }}"
        payload: |
          {
            "text": "⚠️ *LLM Gateway Fallback Alert*: Local inference timed out or failed on `{{ inputs.vllm_model }}`. Execution `<{{ (kestra.url ?? 'http://localhost:8080/') ~ 'ui/' ~ execution.namespace ~ '/executions/' ~ execution.id }}|{{ execution.id }}>` seamlessly failed over to Google Gemini (`{{ inputs.gemini_model }}`)."
          }
    else:
      - id: log_local_success
        type: io.kestra.plugin.core.log.Log
        description: Record zero-cost local inference completion.
        message: "✅ Local vLLM/Ollama inference succeeded with model {{
          inputs.vllm_model }}."

  - id: record_langfuse_trace
    type: io.kestra.plugin.scripts.python.Script
    description: Push unified telemetry, prompt-response logs, and routing metadata
      to Langfuse for LLMOps observability.
    containerImage: python:3.11-slim
    beforeCommands:
      - pip install -q --no-cache-dir "langfuse>=2.0.0"
    env:
      LANGFUSE_PUBLIC_KEY: "{{ secret('LANGFUSE_PUBLIC_KEY') }}"
      LANGFUSE_SECRET_KEY: "{{ secret('LANGFUSE_SECRET_KEY') }}"
      LANGFUSE_HOST: "{{ secret('LANGFUSE_HOST') ?? 'https://cloud.langfuse.com' }}"
      PROMPT: "{{ inputs.prompt }}"
      RESPONSE_TEXT: "{{ outputs.call_gemini_fallback.textOutput ??
        outputs.call_local_vllm.textOutput ?? 'No response returned' }}"
      PROVIDER_USED: "{{ outputs.call_local_vllm.textOutput is defined ? 'ollama_vllm'
        : 'google_gemini' }}"
      FALLBACK_TRIGGERED: "{{ outputs.call_local_vllm.textOutput is not defined ?
        'true' : 'false' }}"
      MODEL_USED: "{{ outputs.call_local_vllm.textOutput is defined ?
        inputs.vllm_model : inputs.gemini_model }}"
      EXECUTION_ID: "{{ execution.id }}"
    script: |
      import os
      import json
      from langfuse import Langfuse

      public_key = os.environ.get("LANGFUSE_PUBLIC_KEY")
      secret_key = os.environ.get("LANGFUSE_SECRET_KEY")
      host = os.environ.get("LANGFUSE_HOST", "https://cloud.langfuse.com")

      langfuse = Langfuse(public_key=public_key, secret_key=secret_key, host=host)

      trace = langfuse.trace(
          name="llm-gateway-routing",
          input=os.environ.get("PROMPT"),
          output=os.environ.get("RESPONSE_TEXT"),
          metadata={
              "provider": os.environ.get("PROVIDER_USED"),
              "fallback_triggered": os.environ.get("FALLBACK_TRIGGERED") == "true",
              "model": os.environ.get("MODEL_USED"),
              "execution_id": os.environ.get("EXECUTION_ID"),
          },
      )
      langfuse.flush()

      trace_url = f"{host.rstrip('/')}/trace/{trace.id}"
      print(f"Langfuse trace recorded: {trace_url}")
      print("::" + json.dumps({"outputs": {"trace_id": trace.id, "trace_url": trace_url}}) + "::")

  - id: gateway_output
    type: io.kestra.plugin.core.output.OutputValues
    description: Expose final response text and routing telemetry as typed Kestra outputs.
    values:
      response: "{{ outputs.call_gemini_fallback.textOutput ??
        outputs.call_local_vllm.textOutput }}"
      provider_used: "{{ outputs.call_local_vllm.textOutput is defined ? 'ollama_vllm'
        : 'google_gemini' }}"
      model_used: "{{ outputs.call_local_vllm.textOutput is defined ?
        inputs.vllm_model : inputs.gemini_model }}"
      fallback_triggered: "{{ outputs.call_local_vllm.textOutput is not defined }}"
      langfuse_trace_url: "{{ outputs.record_langfuse_trace.vars.trace_url }}"

errors:
  - id: alert_gateway_crash
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    description: High-severity alert dispatched if both primary and fallback providers fail.
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {
        "text": "🔥 *Fatal Outage in LLM Gateway* for execution `<{{ (kestra.url ?? 'http://localhost:8080/') ~ 'ui/' ~ execution.namespace ~ '/executions/' ~ execution.id }}|{{ execution.id }}>`. Both local inference and cloud fallback failed."
      }

Running large language models locally using engines like vLLM or Ollama eliminates per-token API charges and keeps proprietary prompts on-premises. However, local GPU nodes are susceptible to CUDA out-of-memory errors, model reloading delays, and unexpected daemon downtime. This blueprint orchestrates an intelligent LLM gateway that routes requests to a local instance first, gracefully fails over to Google Gemini when local capacity degrades, and records unified telemetry in Langfuse for centralized observability.

How it works

An authenticated Webhook trigger (on_request) serves as an HTTP entrypoint for applications, while a disabled-by-default Schedule trigger (nightly_health_check) enables periodic synthetic gateway testing.
call_local_vllm (io.kestra.plugin.ai.completion.ChatCompletion) queries the local inference server using the io.kestra.plugin.ai.provider.Ollama provider. It is configured with allowFailure: true and a strict 15-second execution timeout, ensuring that hung local daemons do not block the pipeline.
check_fallback (io.kestra.plugin.core.flow.If) evaluates whether local inference returned output:
- If local inference failed (outputs.call_local_vllm.textOutput is not defined), call_gemini_fallback (io.kestra.plugin.ai.provider.GoogleGemini) routes the prompt to Google Gemini (gemini-3.1-flash-lite), while notify_slack_fallback (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) informs engineers that failover was engaged.
- If local inference succeeded, log_local_success records a clean confirmation log.
record_langfuse_trace (io.kestra.plugin.scripts.python.Script) runs an isolated Python container that connects to Langfuse, logs prompt and completion tokens, flags whether fallback was triggered, and records the active model name and execution ID.
gateway_output (io.kestra.plugin.core.output.OutputValues) exposes the final text, provider name, and Langfuse trace URL as structured execution variables.
A root-level errors safeguard dispatches an urgent Slack alert if both local and cloud tiers fail simultaneously.

What you get

Zero-token cost baseline: Routes high-volume requests to local hardware whenever available.
Zero-downtime reliability: Automatic failover to Google Gemini handles local server crashes, reboots, or capacity bottlenecks.
Centralized LLMOps observability: Every prompt, response, latency metric, and routing decision is searchable in Langfuse.
Instant incident awareness: Slack notifications alert AI engineers whenever the gateway relies on fallback capacity.

Who it's for

AI Engineers deploying hybrid local/cloud generative AI applications.
Platform Engineers building internal developer LLM gateways and API proxies.
FinOps teams managing and minimizing commercial LLM API expenditure.

Why orchestrate this with Kestra

Building LLM routing logic directly into application code often creates tightly coupled dependencies, unhandled timeout bugs, and fragmented logs. Kestra decouples gateway routing into a declarative YAML workflow. You benefit from automatic retries, timeout enforcement, secret masking via {{ secret(...) }}, execution trace histories, and built-in event-driven webhooks without managing a dedicated reverse-proxy cluster.

Prerequisites

A local Ollama or vLLM server running on your network (accessible from the Kestra worker).
A Google AI Studio API key with access to Gemini 3.1 Flash-Lite.
A Langfuse account (Cloud or self-hosted) with project API keys.

Secrets

Configure the following secrets in your Kestra namespace:

GEMINI_API_KEY: API key for Google Gemini fallback completions.
LANGFUSE_PUBLIC_KEY: Langfuse project public API key.
LANGFUSE_SECRET_KEY: Langfuse project secret API key.
LANGFUSE_HOST: Langfuse host URL (e.g., https://cloud.langfuse.com or self-hosted instance).
SLACK_WEBHOOK_URL: Slack Incoming Webhook endpoint for triage alerts.
WEBHOOK_KEY: Secret authentication key for the gateway webhook trigger.

Quick start

Configure the required secrets in your Kestra namespace settings.
Ensure your local Ollama or vLLM instance is running and has the model pulled (e.g., ollama run llama3.2:3b).
Import this blueprint into your Kestra instance.
Click Execute with the default inputs. If local Ollama is reachable, it generates a response locally; if unreachable, watch Kestra gracefully switch to Gemini and view the resulting trace link in the Outputs tab.
Connect external applications by posting JSON payloads to the gateway webhook URL.

How to extend

Add token bucket rate limiting using Kestra's concurrency limits to prevent overwhelming the local GPU.
Swap Google Gemini for Anthropic Claude or OpenAI GPT-4o as the secondary or tertiary fallback tier.
Integrate Langfuse prompt management to fetch versioned prompt templates dynamically before inference.
Store generated responses and metadata in a vector database or PostgreSQL warehouse for downstream analytics.

Build an LLM Gateway with Local vLLM, Gemini Cloud Fallback, and Langfuse Tracing

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

Build an LLM Gateway with Local vLLM, Gemini Cloud Fallback, and Langfuse Tracing

How it works

What you get

Who it's for

Why orchestrate this with Kestra

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More AI Blueprints

LLM Token and Dollar Budget Guardrail for Teams

Audit LLM Endpoints for Prompt Injections and Canary Leaks

Gate Every Prompt and Model Change Behind a Golden Eval Set

LanceDB Vector Table Fragment Compaction and Storage Sentinel

Monitor LiteLLM Proxy Budgets and Automate Provider Fallback Alerts via Slack

LangChain Adversarial Red-Team and Jailbreak Guardrail

New to Kestra?