LiteLLM Proxy Budget Quota & RPM Circuit Breaker

id: litellm-proxy-budget-and-rpm-fallback-circuit-breaker
namespace: company.ai

description: |
  Audit LiteLLM Proxy admin endpoints, evaluate user spend budgets against spend caps,
  detect HTTP 429 rate limits across model deployments, trip circuit breakers to reroute
  traffic to fallback models, and record active routing state in Kestra KV store.

concurrency:
  limit: 1
  behavior: QUEUE

inputs:
  - id: litellm_proxy_url
    type: STRING
    displayName: LiteLLM Proxy URL
    description: Base URL of the LiteLLM Proxy administration endpoint.
    defaults: "http://localhost:4000"

  - id: user_id
    type: STRING
    displayName: LiteLLM User ID
    description: User or team identifier to audit for spend budget and rate limit
      consumption.
    defaults: "team-data-science"

  - id: primary_model
    type: STRING
    displayName: Primary Model Deployment
    description: Default high-capability LLM deployment prioritized for production
      routing (e.g., Anthropic Claude 3.5 Sonnet).
    defaults: "claude-3-5-sonnet"

  - id: fallback_model
    type: STRING
    displayName: Fallback Model Deployment
    description: First fallback LLM deployment routed when primary deployment
      encounters rate limits or budget thresholds (e.g., OpenAI GPT-4o).
    defaults: "gpt-4o"

  - id: secondary_fallback_model
    type: STRING
    displayName: Emergency Fallback Model Deployment
    description: High-throughput, cost-effective fallback LLM routed during critical
      budget exhaustion (e.g., DeepSeek Chat).
    defaults: "deepseek-chat"

  - id: budget_exhaustion_threshold_pct
    type: FLOAT
    displayName: Budget Warning Threshold (%)
    description: Percentage threshold of spend against max budget (spend /
      max_budget * 100) that trips the cost circuit breaker.
    defaults: 90.0

  - id: rpm_headroom_threshold_pct
    type: FLOAT
    displayName: Minimum RPM Headroom Threshold (%)
    description: Minimum percentage of remaining RPM capacity required before
      tripping the rate limit circuit breaker.
    defaults: 10.0

  - id: fail_on_exhaustion
    type: BOOL
    displayName: Fail on Hard Cap Exhaustion
    description: Terminate workflow with failure status when user spend reaches or
      exceeds 100% of maximum budget cap.
    defaults: false

  - id: slack_channel
    type: STRING
    displayName: Slack Incident Channel
    description: Target Slack channel for circuit breaker state change alerts and
      quota notifications.
    defaults: "#ai-gateway-ops"

triggers:
  - id: periodic_litellm_audit
    type: io.kestra.plugin.core.trigger.Schedule
    cron: "*/5 * * * *"
    disabled: true

  - id: proxy_event_webhook
    type: io.kestra.plugin.core.trigger.Webhook
    key: "{{ secret('LITELLM_WEBHOOK_KEY') }}"

tasks:
  - id: get_user_info
    type: io.kestra.plugin.core.http.Request
    description: Queries LiteLLM Proxy /user/info endpoint for team budget
      allocations, current spend, and active API keys.
    uri: "{{ inputs.litellm_proxy_url }}/user/info?user_id={{ inputs.user_id }}"
    method: GET
    headers:
      Authorization: "Bearer {{ secret('LITELLM_MASTER_KEY') }}"
      Accept: "application/json"
    options:
      allowFailed: true
      timeout:
        readIdleTimeout: PT15S

  - id: get_model_info
    type: io.kestra.plugin.core.http.Request
    description: Queries LiteLLM Proxy /model/info endpoint for configured model
      deployments, throughput limits, and RPM quotas.
    uri: "{{ inputs.litellm_proxy_url }}/model/info"
    method: GET
    headers:
      Authorization: "Bearer {{ secret('LITELLM_MASTER_KEY') }}"
      Accept: "application/json"
    options:
      allowFailed: true
      timeout:
        readIdleTimeout: PT15S

  - id: get_spend_user
    type: io.kestra.plugin.core.http.Request
    description: Queries LiteLLM Proxy /spend/user endpoint for itemized spend
      tracking across active API keys and models.
    uri: "{{ inputs.litellm_proxy_url }}/spend/user?user_id={{ inputs.user_id }}"
    method: GET
    headers:
      Authorization: "Bearer {{ secret('LITELLM_MASTER_KEY') }}"
      Accept: "application/json"
    options:
      allowFailed: true
      timeout:
        readIdleTimeout: PT15S

  - id: probe_primary_model_health
    type: io.kestra.plugin.core.http.Request
    description: Probes LiteLLM Proxy health endpoint or model readiness to detect
      upstream provider rate limits or HTTP 429 errors.
    uri: "{{ inputs.litellm_proxy_url }}/health/services"
    method: GET
    headers:
      Authorization: "Bearer {{ secret('LITELLM_MASTER_KEY') }}"
      Accept: "application/json"
    options:
      allowFailed: true
      timeout:
        readIdleTimeout: PT15S

  - id: evaluate_circuit_breaker
    type: io.kestra.plugin.scripts.python.Script
    description: Analyzes LiteLLM responses, compares spend against max budget,
      verifies RPM limits, and determines active circuit breaker state and
      target routing model.
    containerImage: python:3.11-slim
    beforeCommands:
      - pip install -q kestra
    env:
      USER_INFO_BODY: "{{ (outputs.get_user_info.body ?? {}) | toJson }}"
      USER_INFO_CODE: "{{ outputs.get_user_info.code }}"
      MODEL_INFO_BODY: "{{ (outputs.get_model_info.body ?? {}) | toJson }}"
      MODEL_INFO_CODE: "{{ outputs.get_model_info.code }}"
      SPEND_USER_BODY: "{{ (outputs.get_spend_user.body ?? {}) | toJson }}"
      SPEND_USER_CODE: "{{ outputs.get_spend_user.code }}"
      HEALTH_BODY: "{{ (outputs.probe_primary_model_health.body ?? {}) | toJson }}"
      HEALTH_CODE: "{{ outputs.probe_primary_model_health.code }}"
      PRIMARY_MODEL: "{{ inputs.primary_model }}"
      FALLBACK_MODEL: "{{ inputs.fallback_model }}"
      SECONDARY_FALLBACK_MODEL: "{{ inputs.secondary_fallback_model }}"
      BUDGET_THRESHOLD_PCT: "{{ inputs.budget_exhaustion_threshold_pct }}"
      RPM_HEADROOM_PCT: "{{ inputs.rpm_headroom_threshold_pct }}"
      PREVIOUS_CIRCUIT_STATE: "{{ kv('litellm_circuit_state', errorOnMissing=false) ?? 'CLOSED' }}"
      PREVIOUS_ROUTED_MODEL: "{{ kv('litellm_active_model', errorOnMissing=false) ??
        inputs.primary_model }}"
    script: |
      import json
      import os
      import sys
      from datetime import datetime, timezone

      def parse_json(raw_text):
          if not raw_text:
              return {}
          if isinstance(raw_text, dict):
              return raw_text
          try:
              return json.loads(raw_text)
          except Exception:
              return {}

      user_body = parse_json(os.environ.get("USER_INFO_BODY", "{}"))
      user_code = int(os.environ.get("USER_INFO_CODE", "0") or 0)
      model_body = parse_json(os.environ.get("MODEL_INFO_BODY", "{}"))
      model_code = int(os.environ.get("MODEL_INFO_CODE", "0") or 0)
      spend_body = parse_json(os.environ.get("SPEND_USER_BODY", "{}"))
      spend_code = int(os.environ.get("SPEND_USER_CODE", "0") or 0)
      health_body = parse_json(os.environ.get("HEALTH_BODY", "{}"))
      health_code = int(os.environ.get("HEALTH_CODE", "0") or 0)

      primary_model = os.environ.get("PRIMARY_MODEL", "claude-3-5-sonnet")
      fallback_model = os.environ.get("FALLBACK_MODEL", "gpt-4o")
      secondary_fallback_model = os.environ.get("SECONDARY_FALLBACK_MODEL", "deepseek-chat")
      budget_threshold_pct = float(os.environ.get("BUDGET_THRESHOLD_PCT", "90.0"))
      rpm_headroom_pct = float(os.environ.get("RPM_HEADROOM_PCT", "10.0"))
      previous_circuit_state = os.environ.get("PREVIOUS_CIRCUIT_STATE", "CLOSED").strip() or "CLOSED"
      previous_routed_model = os.environ.get("PREVIOUS_ROUTED_MODEL", primary_model).strip() or primary_model

      # Extract spend and max_budget
      user_info = user_body.get("user_info", user_body) if isinstance(user_body, dict) else {}
      spend_val = user_info.get("spend")
      if spend_val is None:
          spend_val = spend_body.get("total_spend", spend_body.get("spend", 0.0))
      try:
          current_spend = float(spend_val or 0.0)
      except (ValueError, TypeError):
          current_spend = 0.0

      max_budget_val = user_info.get("max_budget")
      try:
          max_budget = float(max_budget_val or 0.0) if max_budget_val is not None else 0.0
      except (ValueError, TypeError):
          max_budget = 0.0

      # Calculate budget utilization
      if max_budget > 0:
          budget_utilization_pct = round((current_spend / max_budget) * 100.0, 2)
      else:
          budget_utilization_pct = 0.0

      # Rate limit detection across probe and admin endpoints
      is_rate_limited = False
      rate_limit_source = "none"
      if health_code == 429:
          is_rate_limited = True
          rate_limit_source = "probe_health_endpoint_429"
      elif user_code == 429 or model_code == 429 or spend_code == 429:
          is_rate_limited = True
          rate_limit_source = "admin_api_endpoint_429"

      # Inspect model deployment RPM headroom from model info
      models_data = model_body.get("data", []) if isinstance(model_body, dict) else []
      primary_rpm_limit = None
      for item in models_data:
          if isinstance(item, dict) and item.get("model_name") == primary_model:
              litellm_params = item.get("litellm_params", {})
              primary_rpm_limit = litellm_params.get("rpm")

      # Inspect active API keys
      active_keys = user_info.get("key_aliases", {})
      key_count = len(active_keys) if isinstance(active_keys, dict) else 0

      # Circuit Breaker Evaluation Logic
      is_budget_hard_cap = (max_budget > 0) and (current_spend >= max_budget)
      is_budget_warning = (max_budget > 0) and (budget_utilization_pct >= budget_threshold_pct)

      if is_budget_hard_cap:
          circuit_state = "OPEN"
          trip_reason = "BUDGET_HARD_CAP_EXHAUSTED"
          active_model = secondary_fallback_model
      elif is_budget_warning:
          circuit_state = "OPEN"
          trip_reason = "BUDGET_WARNING_THRESHOLD_EXCEEDED"
          active_model = fallback_model
      elif is_rate_limited:
          circuit_state = "OPEN"
          trip_reason = f"PRIMARY_PROVIDER_HTTP_429 ({rate_limit_source})"
          active_model = fallback_model
      else:
          if previous_circuit_state == "OPEN":
              circuit_state = "HALF_OPEN"
              trip_reason = "RECOVERY_COOLDOWN_PROBE"
              active_model = primary_model
          else:
              circuit_state = "CLOSED"
              trip_reason = "NOMINAL"
              active_model = primary_model

      metrics_summary = {
          "circuit_state": circuit_state,
          "active_model": active_model,
          "trip_reason": trip_reason,
          "current_spend_usd": current_spend,
          "max_budget_usd": max_budget,
          "budget_utilization_pct": budget_utilization_pct,
          "rate_limit_detected": is_rate_limited,
          "active_key_count": key_count,
          "previous_state": previous_circuit_state,
          "evaluated_at": datetime.now(timezone.utc).isoformat()
      }

      print(f"LiteLLM Circuit Breaker Evaluation:")
      print(f"  State: {circuit_state} (Reason: {trip_reason})")
      print(f"  Active Route: {active_model} (Primary: {primary_model})")
      print(f"  Spend: ${current_spend:.2f} / ${max_budget:.2f} ({budget_utilization_pct:.1f}%)")
      print(f"  Rate Limit Detected: {is_rate_limited}")

      output_payload = {
          "circuit_state": circuit_state,
          "active_model": active_model,
          "trip_reason": trip_reason,
          "current_spend": current_spend,
          "max_budget": max_budget,
          "budget_utilization_pct": budget_utilization_pct,
          "rate_limit_detected": is_rate_limited,
          "metrics_summary": metrics_summary,
          "evaluated_at": metrics_summary["evaluated_at"]
      }

      try:
          from kestra import Kestra
          Kestra.outputs(output_payload)
      except ImportError:
          print("::" + json.dumps({"outputs": output_payload}) + "::")

  - id: set_circuit_state_kv
    type: io.kestra.plugin.core.kv.Set
    description: Updates the circuit breaker state (CLOSED, OPEN, HALF_OPEN) in
      Kestra KV store.
    key: "litellm_circuit_state"
    value: "{{ outputs.evaluate_circuit_breaker.vars.circuit_state }}"
    kvType: STRING

  - id: set_active_model_kv
    type: io.kestra.plugin.core.kv.Set
    description: Updates the active routed LLM model endpoint in Kestra KV store for
      downstream application lookup.
    key: "litellm_active_model"
    value: "{{ outputs.evaluate_circuit_breaker.vars.active_model }}"
    kvType: STRING

  - id: set_spend_metrics_kv
    type: io.kestra.plugin.core.kv.Set
    description: Persists telemetry metrics including current spend, budget limit,
      and rate limit indicators in Kestra KV store.
    key: "litellm_spend_metrics"
    value: "{{ outputs.evaluate_circuit_breaker.vars.metrics_summary | toJson }}"
    kvType: JSON

  - id: check_circuit_trip_alert
    type: io.kestra.plugin.core.flow.If
    description: Evaluates whether the circuit breaker tripped or entered OPEN state
      to dispatch notifications.
    condition: "{{ outputs.evaluate_circuit_breaker.vars.circuit_state == 'OPEN' }}"
    then:
      - id: notify_slack_circuit_trip
        type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
        description: Dispatches incident notification to Slack detailing budget quota
          exhaustion or rate limits with automatic model rerouting.
        url: "{{ secret('SLACK_WEBHOOK_URL') }}"
        payload: |
          {
            "channel": "{{ inputs.slack_channel }}",
            "username": "LiteLLM-Circuit-Breaker",
            "text": "🚨 *ALERT: LiteLLM Circuit Breaker Tripped - Traffic Rerouted!*",
            "blocks": [
              {
                "type": "header",
                "text": {
                  "type": "plain_text",
                  "text": "🚨 LiteLLM Quota Exceeded: Model Fallback Activated"
                }
              },
              {
                "type": "section",
                "fields": [
                  {
                    "type": "mrkdwn",
                    "text": "*User / Team:*\n`{{ inputs.user_id }}`"
                  },
                  {
                    "type": "mrkdwn",
                    "text": "*Circuit State:*\n`{{ outputs.evaluate_circuit_breaker.vars.circuit_state }}`"
                  },
                  {
                    "type": "mrkdwn",
                    "text": "*Trip Reason:*\n`{{ outputs.evaluate_circuit_breaker.vars.trip_reason }}`"
                  },
                  {
                    "type": "mrkdwn",
                    "text": "*Active Route:*\n`{{ outputs.evaluate_circuit_breaker.vars.active_model }}`"
                  },
                  {
                    "type": "mrkdwn",
                    "text": "*Spend / Budget:*\n`${{ outputs.evaluate_circuit_breaker.vars.current_spend }} / ${{ outputs.evaluate_circuit_breaker.vars.max_budget }} ({{ outputs.evaluate_circuit_breaker.vars.budget_utilization_pct }}%)`"
                  },
                  {
                    "type": "mrkdwn",
                    "text": "*Primary Model:*\n`{{ inputs.primary_model }}`"
                  }
                ]
              },
              {
                "type": "context",
                "elements": [
                  {
                    "type": "mrkdwn",
                    "text": "Routing state recorded in Kestra KV store under `litellm_active_model`. Downstream consumers redirected to fallback model endpoint."
                  }
                ]
              }
            ]
          }

      - id: check_hard_cap_exhaustion
        type: io.kestra.plugin.core.flow.If
        description: Evaluates whether to halt execution with failure if
          fail_on_exhaustion is enabled and hard cap is breached.
        condition: "{{ inputs.fail_on_exhaustion and
          outputs.evaluate_circuit_breaker.vars.trip_reason ==
          'BUDGET_HARD_CAP_EXHAUSTED' }}"
        then:
          - id: fail_on_hard_cap
            type: io.kestra.plugin.core.execution.Fail
            description: Halts workflow with failure to enforce budget cap protection.

    else:
      - id: log_nominal_state
        type: io.kestra.plugin.core.log.Log
        description: Records healthy operating metrics when primary model deployment is
          operational within quota boundaries.
        message: "[NOMINAL] LiteLLM Proxy circuit breaker is CLOSED. Active route: {{
          outputs.evaluate_circuit_breaker.vars.active_model }} (Spend: ${{
          outputs.evaluate_circuit_breaker.vars.current_spend }} / ${{
          outputs.evaluate_circuit_breaker.vars.max_budget }}, Utilization: {{
          outputs.evaluate_circuit_breaker.vars.budget_utilization_pct }}%, Rate
          Limited: {{ outputs.evaluate_circuit_breaker.vars.rate_limit_detected
          }})."

  - id: export_circuit_breaker_telemetry
    type: io.kestra.plugin.core.output.OutputValues
    description: Exposes telemetry variables to execution outputs for downstream
      workflows or dashboard consumption.
    values:
      circuit_state: "{{ outputs.evaluate_circuit_breaker.vars.circuit_state }}"
      active_model: "{{ outputs.evaluate_circuit_breaker.vars.active_model }}"
      trip_reason: "{{ outputs.evaluate_circuit_breaker.vars.trip_reason }}"
      current_spend_usd: "{{ outputs.evaluate_circuit_breaker.vars.current_spend }}"
      max_budget_usd: "{{ outputs.evaluate_circuit_breaker.vars.max_budget }}"
      budget_utilization_pct: "{{ outputs.evaluate_circuit_breaker.vars.budget_utilization_pct }}"
      rate_limit_detected: "{{ outputs.evaluate_circuit_breaker.vars.rate_limit_detected }}"
      evaluated_at: "{{ outputs.evaluate_circuit_breaker.vars.evaluated_at }}"

errors:
  - id: notify_circuit_breaker_error
    type: io.kestra.plugin.slack.notifications.SlackIncomingWebhook
    description: Dispatches high-priority Slack notification if the LiteLLM circuit
      breaker audit workflow encounters an unhandled execution error.
    url: "{{ secret('SLACK_WEBHOOK_URL') }}"
    payload: |
      {
        "channel": "{{ inputs.slack_channel }}",
        "username": "LiteLLM-Circuit-Breaker-Error",
        "text": "🚨 *CRITICAL: LiteLLM Circuit Breaker Audit Failed!*\n• *Execution:* `{{ execution.id }}`\n• *Flow:* `{{ flow.id }}`\n• *Namespace:* `{{ flow.namespace }}`\n• *LiteLLM Host:* `{{ inputs.litellm_proxy_url }}`\n• *Action Required:* Check Kestra logs for connection or parsing failures."
      }

LiteLLM Proxy is an open-source OpenAI-compatible gateway that centralizes authentication, load balancing, cost tracking, and rate limits across more than 100 large language models (such as Anthropic Claude, OpenAI GPT-4o, and DeepSeek).

Production AI architectures require automated safeguards against sudden token cost surges and upstream provider rate limit spikes. When an engineering team exhausts its allocated monthly budget or encounters HTTP 429 rate limit rejections from foundational model providers, downstream client applications experience cascading timeouts and degraded user experiences.

This blueprint provides an automated circuit breaker and FinOps quota controller that polls LiteLLM Proxy admin endpoints, audits active user key spend against assigned caps, detects rate limit errors, immediately trips the circuit to OPEN, switches traffic to fallback model endpoints, updates Kestra KV store routing state, and dispatches incident alerts to Slack.

How it works

Admin Endpoint Polling: Queries LiteLLM Proxy administration endpoints (/user/info, /model/info, /spend/user, and /health/services) via io.kestra.plugin.core.http.Request with allowFailed: true to prevent workflow crashes during partial API degradation.
Quota & Rate Limit Evaluation: An isolated Python script calculates budget consumption percentages (spend / max_budget * 100), checks for HTTP 429 status codes, and audits model deployment requests-per-minute (RPM) limits.
Circuit Breaker State Machine:
- CLOSED: Normal operation; routes traffic to primary_model (e.g., claude-3-5-sonnet).
- OPEN (Budget Warning): Spend reaches or exceeds 90% of budget cap; reroutes traffic to fallback_model (e.g., gpt-4o).
- OPEN (Hard Cap Exhausted): Spend reaches 100% of budget; switches routing to high-throughput low-cost model (e.g., deepseek-chat).
- OPEN (Rate Limited): Primary provider returns HTTP 429; trips circuit breaker and switches traffic to fallback_model.
- HALF_OPEN: Probes recovery during subsequent scheduled execution cycles once error conditions clear.
KV Store State Persistence: Writes litellm_circuit_state, litellm_active_model, and structured spend telemetry into Kestra KV store using io.kestra.plugin.core.kv.Set. Downstream microservices and Kestra subflows read these values to direct prompt requests to healthy endpoints.
Slack Incident Alerting: Dispatches actionable Slack alerts via io.kestra.plugin.slack.notifications.SlackIncomingWebhook with spend figures, active routing targets, and root-cause indicators.

Architecture diagram

flowchart TD
    A[Schedule or Webhook Trigger] --> B[get_user_info HTTP]
    A --> C[get_model_info HTTP]
    A --> D[get_spend_user HTTP]
    A --> E[probe_primary_model_health HTTP]
    B --> F[evaluate_circuit_breaker Python Script]
    C --> F
    D --> F
    E --> F
    F --> G[set_circuit_state_kv KV Set]
    F --> H[set_active_model_kv KV Set]
    F --> I[set_spend_metrics_kv KV Set]
    G --> J{check_circuit_trip_alert Flowable If}
    H --> J
    I --> J
    J -- Circuit OPEN --> K[notify_slack_circuit_trip Slack Webhook]
    K --> L{check_hard_cap_exhaustion Flowable If}
    L -- Hard Cap Breached --> M[fail_on_hard_cap Fail]
    L -- Normal Fallback --> N[export_circuit_breaker_telemetry OutputValues]
    J -- Circuit CLOSED --> O[log_nominal_state Log]
    O --> N

Architecture components

HTTP Polling Tasks (get_user_info, get_model_info, get_spend_user, probe_primary_model_health): Collect real-time spend, key allocation, model metadata, and health signals from the LiteLLM Proxy admin API.
Circuit Evaluation Engine (evaluate_circuit_breaker): Deterministic Python script that evaluates budget thresholds, detects 429 status codes, and transitions circuit breaker state.
KV Store Tasks (set_circuit_state_kv, set_active_model_kv, set_spend_metrics_kv): Persist state in Kestra KV store to decouple gateway monitoring from client application routing.
Notification & Protection Gate (check_circuit_trip_alert): Dispatches Slack alerts when circuit state changes and optionally halts executions when hard spending caps are reached.
Output Manifest (export_circuit_breaker_telemetry): Emits structured JSON outputs for upstream orchestrators and monitoring dashboards.

Key capabilities

Multi-Model Fallback Routing: Dynamically directs downstream calls across Anthropic Claude, OpenAI GPT-4o, and DeepSeek deployments based on real-time availability and spend.
Proactive Budget Guardrails: Flags budget exhaustion before total cut-off occurs, allowing graceful degradation rather than hard application failure.
Zero-Downtime KV State Management: Decouples monitoring frequency from runtime inference queries by using Kestra KV store as a centralized routing source of truth.
Dual Trigger Support: Runs on a continuous 5-minute schedule or receives inbound webhook events from LiteLLM proxy event hooks.

What you get

Automated detection of LiteLLM budget exhaustion and provider rate limits.
Instant fallback routing across configured model tiers (claude-3-5-sonnet -> gpt-4o -> deepseek-chat).
Centralized routing state and spend metrics published to Kestra KV store.
Formatted Slack incident alerts with spend figures and root-cause summaries.
Configurable circuit breaker thresholds for early warning and hard quota enforcement.

Who it's for

Platform Engineers managing enterprise LLM gateway infrastructure and API keys.
FinOps Practitioners enforcing departmental AI spend quotas and cost caps.
AI / MLOps Engineers requiring automated failover across competing foundational model providers.

Why orchestrate this with Kestra

Native KV Store Integration: Stores circuit state and active model endpoints directly in Kestra namespace KV store without requiring external Redis or DynamoDB instances.
Reliable Secret Management: Injects LITELLM_MASTER_KEY and SLACK_WEBHOOK_URL securely at runtime via encrypted secrets.
Built-in Resilience: Employs allowFailed: true and workflow-level errors handlers to prevent transient network issues from crashing the control loop.
Declarative Concurrency: Limits concurrent evaluations using concurrency.limit: 1 and behavior: QUEUE to prevent race conditions during circuit state updates.

Inputs

Name	Type	Default	Description
`litellm_proxy_url`	`STRING`	`"http://localhost:4000"`	Base URL of the LiteLLM Proxy administration endpoint.
`user_id`	`STRING`	`"team-data-science"`	Target user ID or team identifier to audit for spend and rate limits.
`primary_model`	`STRING`	`"claude-3-5-sonnet"`	Default LLM deployment prioritized for production routing.
`fallback_model`	`STRING`	`"gpt-4o"`	First fallback LLM deployment routed during primary rate limit or budget warning.
`secondary_fallback_model`	`STRING`	`"deepseek-chat"`	High-throughput, cost-effective fallback LLM routed during hard budget exhaustion.
`budget_exhaustion_threshold_pct`	`FLOAT`	`90.0`	Percentage of budget consumption that trips the warning circuit breaker.
`rpm_headroom_threshold_pct`	`FLOAT`	`10.0`	Minimum percentage of remaining RPM capacity required before tripping breaker.
`fail_on_exhaustion`	`BOOL`	`false`	Raise an execution failure when user spend reaches or exceeds 100% of budget cap.
`slack_channel`	`STRING`	`"#ai-gateway-ops"`	Target Slack channel for incident alerts and routing notifications.

Expected outputs

{{ outputs.export_circuit_breaker_telemetry.values.circuit_state }}: Current circuit breaker state (CLOSED, OPEN, HALF_OPEN).
{{ outputs.export_circuit_breaker_telemetry.values.active_model }}: Active routed model deployment (e.g., claude-3-5-sonnet, gpt-4o, deepseek-chat).
{{ outputs.export_circuit_breaker_telemetry.values.trip_reason }}: Explanation for current circuit breaker state.
{{ outputs.export_circuit_breaker_telemetry.values.current_spend_usd }}: Current cumulative spend for the target user in USD.
{{ outputs.export_circuit_breaker_telemetry.values.max_budget_usd }}: Maximum budget limit assigned to the user in USD.
{{ outputs.export_circuit_breaker_telemetry.values.budget_utilization_pct }}: Percentage of maximum budget consumed.
{{ outputs.export_circuit_breaker_telemetry.values.rate_limit_detected }}: Boolean flag indicating whether HTTP 429 was detected.
{{ outputs.export_circuit_breaker_telemetry.values.evaluated_at }}: ISO-8601 UTC timestamp of the circuit evaluation.

Prerequisites

LiteLLM Proxy instance running with Admin API enabled and accessible over HTTP/HTTPS.
LiteLLM Master Key configured with administrative permissions for /user/info and /model/info.
Slack Incoming Webhook URL configured for incident alerts.

Secrets

LITELLM_MASTER_KEY: Master API key for authenticating against LiteLLM Proxy admin endpoints.
LITELLM_WEBHOOK_KEY: Secret authentication key for the inbound webhook trigger.
SLACK_WEBHOOK_URL: Webhook URL for dispatching incident notifications to Slack.

Quick start

Configure secrets LITELLM_MASTER_KEY, LITELLM_WEBHOOK_KEY, and SLACK_WEBHOOK_URL in your Kestra namespace.
Import this blueprint into your Kestra instance under namespace company.ai.
Set litellm_proxy_url to your LiteLLM Proxy gateway address.
Run a manual execution to audit current spend and initialize litellm_active_model in the KV store.
Enable the periodic_litellm_audit schedule trigger to establish automated 5-minute monitoring.

How to extend

Automated LiteLLM Config Patching: Add an io.kestra.plugin.core.http.Request task using POST /model/fallbacks to dynamically update LiteLLM router fallbacks directly inside the proxy daemon.
Multi-Tenant Team Iteration: Wrap the inspection tasks inside an io.kestra.plugin.core.flow.Loop loop to evaluate spend caps across all active LiteLLM user accounts in a single run.
PagerDuty Incident Escalation: Integrate io.kestra.plugin.pagerduty.notifications.PagerDutyAlert when hard budget exhaustion occurs to alert on-call engineering leads.

LiteLLM Proxy Budget Quota & RPM Fallback Circuit Breaker

How it works

Architecture diagram

Architecture components

Key capabilities

What you get

Who it's for

Why orchestrate this with Kestra

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

New to Kestra?

LiteLLM Proxy Budget Quota & RPM Fallback Circuit Breaker

How it works

Architecture diagram

Architecture components

Key capabilities

What you get

Who it's for

Why orchestrate this with Kestra

Inputs

Expected outputs

Prerequisites

Secrets

Quick start

How to extend

Links

Tasks

Orchestrate with Kestra

Share this Blueprint

Create a Blueprint

More AI Blueprints

Gate LLM Judge Drift Before It Silently Changes Verdicts

Langfuse Daily LLM Trace Quality and Cost Sentinel

LLM Model Retirement Migrator for OpenAI, Anthropic and Gemini Model IDs in Your Repositories

LLM Token and Dollar Budget Guardrail for Teams

LangSmith & OpenTelemetry LLM Token Spend and Latency Outlier Gate

LLM Evaluation & Regression Testing

More Cloud Blueprints

Kubernetes ConfigMap and Secret Hash Drift Sentinel

Kubecost Namespace Rightsizing with Approval and Chargeback

KMS Key Rotation Compliance Gate

Kubernetes CronJob and Batch Job Execution Failure Sentinel

Kubernetes API Server Request Latency Spike Sentinel

Kubernetes CNI IPAM Node Pod Subnet Exhaustion Sentinel

New to Kestra?