Keep Gemini Classification Running Through a Model Outage With Retry and Fallback

Preview-tier LLM APIs go UNAVAILABLE under load, and a webhook-triggered classification flow cannot just wait for the provider to recover. This blueprint keeps an inbound classification pipeline running when Gemini returns a 503 by retrying transient failures, falling back across three model versions in order, capping concurrent runs so the flow does not blow through its own Gemini quota, and posting one Slack message whenever the fallback path actually fires.

How it works

1. The on_request trigger (io.kestra.plugin.core.trigger.Webhook) starts one execution per inbound request, exposing the payload as trigger.body.
2. classify_request (io.kestra.plugin.gemini.ChatCompletion) calls gemini-3.1-pro-preview first, with a retry policy of 5 attempts on a 1-second constant interval and allowFailure: true so a persistent failure does not kill the execution outright.
3. classify_fallback_1 calls gemini-3-pro-preview, gated by runIf so it only runs when the first call produced no prediction.
4. classify_fallback_2 calls gemini-3.5-flash-lite, gated by a runIf that checks both prior tasks failed to produce a prediction.
5. notify_if_fallback_used (io.kestra.plugin.slack.notifications.SlackIncomingWebhook) fires only when a fallback task actually ran, giving the team visibility into outage cadence without paging on every retry.
6. A flow-level concurrency block (behavior: QUEUE, limit: 10) caps simultaneous executions so a burst of inbound requests cannot exhaust the Gemini quota or hammer the fallback chain in parallel.

What you get

• Automatic recovery from transient 503s without a single line of retry code.
• A three-model fallback chain that trades a few points of accuracy for continued uptime during an outage.
• Concurrency-capped execution so a traffic spike cannot exceed your own API quota.
• One Slack signal per outage event instead of a page for every retry.

Who it's for

• Platform and AI engineering teams running Gemini classification or routing in production.
• Teams who have been paged for a model provider outage and want a durable default instead of a one-off patch.
• Anyone chaining a Gemini call into a webhook-triggered pipeline that cannot simply wait for the provider to recover.

Why orchestrate this with Kestra

A hand-rolled script can retry once and catch an exception, but keeping retry, allowFailure, and concurrency consistent across every task as the flow grows is where glue code drifts. Kestra applies retry and fallback declaratively per task, exposes runIf to gate each fallback on the exact prior outputs, and enforces the concurrency queue at the flow level so the guarantee holds no matter how many tasks get added later. Every attempt, retry, and fallback is visible in one execution's history.

Prerequisites

• A Google Gemini API key with access to at least two model versions for fallback.
• A webhook source posting subject and message fields.
• A Slack incoming webhook for the fallback alert.

Secrets

• GEMINI_API_KEY: API key shared by all three classification calls.
• WEBHOOK_KEY: the signing key for the inbound Webhook trigger.
• SLACK_WEBHOOK: incoming webhook URL for the fallback notification.

Quick start

1. Add the secrets above to your Kestra namespace.
2. Point the webhook source at .../executions/webhook/company.team/gemini-model-fallback-resilience/{{ secret('WEBHOOK_KEY') }}.
3. Send a test request and confirm classify_request succeeds under normal conditions.
4. Temporarily set an invalid model on classify_request to confirm the fallback chain and Slack alert fire correctly.

How to extend

• Add a fourth fallback to a different provider's plugin for a cross-vendor safety net.
• Log the fallback cadence to a database to track how often each model version goes UNAVAILABLE.
• Replace the webhook trigger with a queue-based trigger for higher-volume sources.
• Swap the Slack alert for PagerDuty when the fallback chain exhausts all three models.

Links

• plugin-gemini docs
• Retries in Kestra