Retry and Fallback: A Minimal Implementation That Solves 90% of LLM API Failures

Anyone building an independent product has probably experienced nights like this: an upstream LLM API intermittently times out, your service freezes for 30 seconds before throwing an error, and the user simply closes the page. When you check the logs, you find this kind of jitter happens dozens of times a day, and each incident requires manual intervention to recover.

Our team once tallied up our own handling workflow: from noticing the error, logging into the server to check logs, manually switching to a backup model, to restoring service — each incident took an average of 25 minutes. Over a month, handling these "non-fatal but annoying" failures ate up more than a dozen hours. Later, we wrapped error retry and fallback into a unified layer; the same problems now recover automatically, with virtually zero manual intervention. This article hands you the minimal viable version of that layer in its entirety.

First, Get Your Account and Key Ready

We'll use ThisToken.AI's OpenAI-compatible API as an example (it aggregates multiple models behind a single API, making it well-suited for multi-model fallback). Registration process:

1. Open https://api.thistoken.ai/register , register with your email and complete verification;
2. Go to the console, create a Key on the "API Keys" page, and copy it somewhere safe (it's only shown once);
3. Top up your balance. Please refer to the official pricing page for rates — we won't quote any numbers here;
4. Note the API endpoint: https://api.thistoken.ai/v1.

We recommend putting the Key in the environment variable THISTOKEN_API_KEY and never committing it to your code repository — a lesson we learned through three incidents.

Core Idea: Retries Solve Jitter, Fallback Solves Unavailability

Two categories of errors should be handled differently:

• Retryable errors: 429 (rate limiting), 5xx, network timeouts. Wait a short while and try again — it will most likely succeed;
• Non-retryable errors: 401 (bad key), 400 (bad parameters). Retrying ten thousand times won't help. You should fall back to a backup model immediately, or fail fast with a clear message to the user.

The key is the backoff strategy: don't retry back-to-back immediately. Use exponential backoff with random jitter, to avoid piling onto an upstream that's already gasping for air.

Code You Can Run Directly

The Python below is our minimal implementation, depending on the official openai SDK (ThisToken.AI is compatible with its protocol):

import os
import time
import random
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["THISTOKEN_API_KEY"],
    base_url="https://api.thistoken.ai/v1",
)

# 模型降级链：主模型失败后，依次尝试备选
MODEL_CHAIN = ["gpt-4o", "gpt-4o-mini", "gpt-3.5-turbo"]

RETRYABLE_STATUS = {408, 409, 429, 500, 502, 503, 504}


def is_retryable(exc) -> bool:
    status = getattr(exc, "status_code", None)
    if status is not None:
        return status in RETRYABLE_STATUS
    return isinstance(exc, (ConnectionError, TimeoutError))


def chat_with_fallback(prompt: str, max_retries: int = 3):
    last_exc = None
    for model in MODEL_CHAIN:
        for attempt in range(max_retries):
            try:
                resp = client.chat.completions.create(
                    model=model,
                    messages=[{"role": "user", "content": prompt}],
                    timeout=15,
                )
                return model, resp.choices[0].message.content
            except Exception as exc:
                last_exc = exc
                if is_retryable(exc) and attempt < max_retries - 1:
                    # 指数退避 + 随机抖动：1s、2s、4s 上下浮动
                    delay = (2 ** attempt) + random.uniform(0, 0.5)
                    print(f"[{model}] 第{attempt+1}次失败，{delay:.1f}s 后重试")
                    time.sleep(delay)
                elif is_retryable(exc):
                    print(f"[{model}] 重试耗尽，切换下一模型")
                    break
                else:
                    # 不可重试错误：直接换模型，别浪费时间
                    print(f"[{model}] 不可重试错误，切换下一模型")
                    break
    raise last_exc


if __name__ == "__main__":
    model, answer = chat_with_fallback("用一句话解释什么是指数退避")
    print(f"实际使用模型: {model}\n{answer}")

Get this code running first, then wire it into your business logic. Here's our before/after comparison: average failure recovery time dropped from about 25 minutes to nearly zero (fully automatic), and the user-facing error rate went from "guaranteed on every jitter event" to "only when the entire fallback chain is down." Assuming about 30 jitter events per month, that saves over 10 hours of ops time — not even counting the hidden cost of user churn.

Three Common Pitfalls

Don't blindly retry non-idempotent operations. For interfaces like order compensation or billing, retries may cause duplicate execution. Add idempotency keys at the business layer.

Always set the timeout explicitly. Many SDKs default to no timeout or extremely long timeouts; a single hang means at least 30 seconds of waiting. Adjust the timeout=15 in the code above to fit your scenario.

Fallback models must have a capability floor. Backup models may be cheaper but also weaker. For critical business flows, we recommend recording which model was actually used in the response (the code already returns it), to make it easy to assess quality during fallback periods.

Next Steps

Once this is running, you can keep evolving it: pipe retry counters into monitoring alerts, differentiate alert severity by error type, and configure different fallback chains for different business lines. But the first step is always to get these few dozen lines of code running — it may well be the highest ROI engineering change you ever make.

If you haven't registered yet, start here: https://api.thistoken.ai/register

---

Ready to try it yourself? Sign up at https://api.thistoken.ai/register to get your API key and start building.