## A Real-World Failure Story

A Real-World Failure Story

Indie developer Kobayashi took on a gig: build an online Q&A assistant for a pet shop, letting owners get free advice on questions like "my cat is vomiting, what should I do" after spending money at the store. He shipped the first version in a single weekend—after all, it's just one API call to a large model, throw the user's question in, a few lines of code.

Two weeks later, the pet shop owner threw the chat logs at him:

• A user asked "can a three-month-old kitten eat adult cat food," and the model replied with two long paragraphs analyzing cat food nutrition, with no conclusion;
• A user asked "my dog accidentally ate chocolate," and the model leisurely responded with a science lecture, without conveying the urgency of inducing vomiting and seeking veterinary care;
• During peak hours, dozens of people asked questions simultaneously, token consumption was staggering, and the end-of-month bill exceeded the development fee the pet shop paid;
• The worst incident: the model gave a specific medication dosage recommendation for a rare symptom, the user followed it and the dog had problems—the owner was nearly hit with a complaint.

Kobayashi's problem wasn't poor code quality—it was that he treated a high-risk, high-frequency scenario like pet healthcare as "one API call." Below, we'll break down these common failure patterns through counterexamples, then lay out a correct path to production.

Four Common Failure Patterns

Failure pattern one: one large model to rule them all. All questions—whether "how often should a cat be dewormed" or "my dog is convulsing, suspected epilepsy"—go through the same large model with the same prompt. The result: simple questions waste money, while urgent questions get no special handling.

Failure pattern two: no risk tiering. The fundamental risk in medical Q&A is asymmetric—a wrong answer in casual chat goes unnoticed, but a wrong answer in an emergency causes real harm. Without tiering, every question effectively carries the same (nonexistent) safety process.

Failure pattern three: if the model is down, so are you. Connecting to only one model vendor means that when that service gets rate-limited or goes down one day, the entire assistant goes mute, and users will simply think "this product is broken."

Failure pattern four: hardcoding multiple SDKs. After realizing the need for multiple models, you import three or four vendors' SDKs into your code, each with different parameter formats, streaming interfaces, and error codes. When a model upgrades, you modify three files; switch a vendor, and you rewrite half a module. Maintenance cost grows linearly with the number of vendors—something small teams can least afford.

The Right Path: Layered Models + Three-Tier Fallback

Architecture Design

The overall design has three layers:

Layer 1: Intent classification and risk grading (lightweight model). Use a cheap, fast small model (with a rule engine as fallback) to do three things with user input: identify intent (daily care / symptom consultation / emergency / out-of-scope chat), extract key entities (pet species, age, symptoms), and assign risk labels. This layer costs roughly a few dozen times less than a large model.

Layer 2: Route to different handling strategies.

• Daily care → lightweight model + fixed knowledge base templates; answering questions about deworming schedules or vaccination timelines doesn't require a flagship model at all;
• Symptom consultation → strong model + pet medical knowledge base (RAG), with prompts that mandate "provide observation advice + clear criteria for seeking veterinary care," and prohibit giving specific medication dosages;
• Emergencies (keyword hits like poisoning from accidental ingestion, trauma, convulsions) → short-circuit directly to preset first-aid guidance scripts + nearest pet hospital information, bypassing model generation entirely; templates guarantee zero hallucination and zero latency;
• Out-of-scope questions → politely decline and steer back to topic.

Layer 3: Fallback chain. Primary model fails or times out → automatically switch to backup model → backup also fails → return a preset degraded message ("System is busy; for emergencies please contact a pet hospital directly") with the hospital's phone number attached. Users never see an error.

Why a Unified AI API Gateway Reduces Maintenance Cost

The key to this architecture is: layered routing means you need to manage multiple models and multiple vendors simultaneously. If you integrate each vendor's SDK directly into your business code, you'll run into three headaches:

1. Inconsistent interfaces: each vendor has different authentication methods, message formats, and streaming response structures—switching models = rewriting the call layer;
2. Scattered keys: multiple API Keys spread across code and environment variables; rotating a key means updating it in several places;
3. Fragmented monitoring: to see "how much each layer spent today and what the failure rates are," you have to log into several backends separately and stitch the data together.

With a unified AI API gateway (such as aggregation gateways like Thistoken), your business code only integrates one OpenAI-compatible format, and switching the underlying model is just changing one model name parameter. The fallback logic for primary-backup switching gets simplified from "if-else across vendor SDKs" to changing one line of configuration; you only maintain one key; usage and costs are visible in a single backend dashboard. For small teams, this doesn't just save one round of development time—it saves the repeated rework on every model iteration and every vendor price change. Maintenance cost goes from "multiplied by the number of vendors" to "essentially constant."

Key Implementation Steps and Process Checklist

1. Preparation
   □ Collect 50-100 real high-frequency questions, manually annotate intent and risk level
   □ Build an emergency keyword list (accidental ingestion, convulsions, heavy bleeding, difficulty breathing…)

2. Layered implementation
   □ Connect to a unified gateway; keep only one call wrapper in business code
   □ Layer 1: lightweight model for classification (low temperature, JSON output)
   □ On classification failure, degrade to regex keyword matching so routing never breaks
   □ Layer 2: dispatch to corresponding models and prompt templates per the routing table
   □ Emergency questions take the template short-circuit, never entering the model

3. Fallback and safety
   □ Primary model timeout of 5 seconds → automatic switch to backup model
   □ Double failure → return degraded message + veterinary guidance
   □ Append a fixed disclaimer and medical advice notice to the end of all symptom-related answers
   □ Add sensitive word filtering at the output layer to block answers with specific medication dosages

4. Launch and observability
   □ Gradual rollout; manually review 10% of conversations for the first two weeks
   □ Monitor call volume, cost, and failure rate per layer in the gateway dashboard
   □ Optimize the keyword list monthly based on classification accuracy data

Results

After Kobayashi rebuilt with this approach: daily care questions made up over 60% of total volume, all routed to the lightweight model, cutting overall token costs to one-third of the original; emergency responses went from over ten seconds to instant template replies; and when one vendor got rate-limited late at night, the fallback chain switched automatically with zero user-visible impact. Most importantly, the pet shop owner never again received complaints like "the AI told me to medicate my dog myself"—because the output layer simply doesn't allow such content through.

Final Thoughts

In scenarios like pet medical Q&A, the competition isn't about who has the bigger model—it's about who has more accurate layering and more reliable fallbacks. If you're about to start, the first step can be connecting to a unified gateway—consolidating model calls in one place lays the foundation for the layered routing, cost monitoring, and primary-backup switching that follow. You can register for a trial at https://api.thistoken.ai/register and get your first-layer classification model up and running.

---

Tired of juggling provider integrations? Register at https://api.thistoken.ai/register and call every model through one base_url.