Background: A "Good Enough" Model Is Quietly Eating Your Profits
When many small teams build internal Q&A chatbots for their companies, the approach is simple: pick the most capable model and throw every question at it. The logic is "the chatbot is a core feature, so we can't cut corners."
That's exactly what our team did at first. For the first two months after our internal knowledge base Q&A went live, all requests went through the same high-end model. It wasn't until our monthly review, when we pulled the log data, that we discovered an awkward fact:
About 78% of questions were high-frequency, simple ones like "How do I file an expense report?", "What's the WiFi password?", or "Where can I check my remaining annual leave?" The answer quality for these questions was virtually indistinguishable between the most expensive model and a mid-tier model—but the difference in the bill was very real.
Even more painful was the time cost: every day, about 40+ questions needed manual reprocessing due to queuing, timeouts, or context length limits, with ops colleagues spending an average of 30-40 minutes per day on them. This time doesn't show up on the API bill, but it's real labor cost.
Core Idea: Split by Scenario, Not by "Gut Feeling"
We eventually split the chatbot's requests into three scenarios, each with a different model strategy. The split wasn't based on model leaderboards, but on question complexity and cost sensitivity.
Scenario 1: High-Frequency Standard Q&A (about 70-80% of traffic)
Typical questions: policy lookups, process guidance, common troubleshooting.
Characteristics: answers can almost always be found in knowledge base documents, questions are phrased relatively consistently, and reasoning requirements are low.
Model strategy: a lightweight, fast-responding model + a smaller max_tokens limit.
We set both the thinking budget and output length very conservatively for this tier. After the switch, the per-request cost for this segment dropped to roughly one quarter of the original, while the "inaccurate answer" rate in user satisfaction surveys stayed unchanged.
Scenario 2: Multi-Document Synthesis and Long-Context Understanding (about 15-25% of traffic)
Typical question: "What are the standards for cross-department business trip approval and expense reimbursement respectively? Please put them into a comparison table."
These questions require the model to read multiple documents and synthesize the information, placing real demands on long-context capability and instruction following.
Model strategy: a mid-to-high-end model, paired with a larger context window and a more refined system prompt.
We didn't cut costs here. This proved to be right: in early attempts to force lightweight models to handle these questions, a single incoherent answer would trigger two or three rounds of follow-ups from users, resulting in higher total token consumption and a worse experience.
Scenario 3: Fallback and Human Handoff (about 5% of traffic)
Typical questions: vague inquiries, questions outside the knowledge base's scope, clearly emotional complaints.
Model strategy: the most capable model for intent recognition and fallback answers; when recognition fails, generate a ticket summary and hand off to a human.
This segment is small, but it determines users' "baseline impression" of the entire chatbot—it's worth using the best model.
Before vs. After the Split
| Dimension | Before (single high-end model) | After (per-scenario models) |
|---|---|---|
| Scenario coverage | 100% of traffic through one model | Dedicated strategy per scenario |
| Cost per simple Q&A | Baseline 100% | ~25% |
| Average first-token latency for simple questions | Baseline | Significantly shorter (lightweight models infer faster) |
| Daily manual interventions | 40+ | ~10 |
| Daily ops handling time | ~30-40 minutes | ~10 minutes |
| Overall monthly API bill | Baseline 100% | ~40% |
| Answer quality for complex questions | Good | Equal or better (prompts tuned per scenario) |
Back-of-the-envelope math: API costs dropped by about 60%, ops saves about half an hour per day, which adds up to roughly half a person-day per month that can go into feature iteration. For a team of two or three people, this "invisible income" is worth more than the bill itself.
Key Infrastructure: Why You Need a Unified Gateway
The plan sounds great, but there's a practical issue in implementation: the three scenarios may involve models from two or three different providers. If each scenario connects directly to each provider's API, you'll run into—
- Three sets of auth, retry, and logging code: rewriting the glue logic every time you add a model;
- Inconsistent API formats: switching models means changing the request body structure, with high regression testing costs;
- No fallback when a model is discontinued or rate-limited: if a provider's model suddenly goes down, the corresponding scenario collapses.
Our approach was to route all requests through a single gateway layer. The benefits show up most directly in three things:
- Switching models is a one-line config change. Scenario 1 routes to model A; one day model A raises prices or degrades, and you switch to model B by changing a model name—zero code changes. Once, due to rate limiting on the provider's side, we switched Scenario 1's model within ten minutes with no user-visible impact.
- Unified billing and usage dashboard. Cost, latency, and error rates for all three scenarios are compared in a single panel. Our monthly "model strategy review" takes half an hour instead of exporting three providers' bills and stitching them together by hand.
- Independent rate-limiting and degradation policies per scenario. Scenario 1 has high volume but high fault tolerance, so it can degrade aggressively; Scenario 3 is the fallback, so it gets the highest priority. This kind of fine-grained control is nearly impossible with direct connections.
The essence of the gateway's value is: it downgrades "which model to choose" from an architectural decision to a configuration item. With the model market changing this fast, this flexibility is survival-level for small teams.
Recommended Rollout Path
If you want to replicate this approach, follow this order for minimal risk:
- First, pull two weeks of historical question logs and do a simple scenario distribution analysis—your distribution will likely differ from our 75/20/5;
- Switch the largest scenario to a lightweight model first and observe answer quality metrics for a week; this segment offers the biggest gains at the lowest risk;
- Handle the fallback scenario last, and polish the human-handoff ticket summary workflow.
The whole process doesn't require rewriting your chatbot, provided you already have (or can easily integrate) a unified gateway that supports multi-model routing.
If you haven't built this layer of infrastructure yet, check out the one we're currently using: sign up and start using it right away, with unified multi-model access and per-scenario routing, config changes taking effect in real time—well suited for teams that want to validate the "per-scenario model switching" playbook at low cost first → https://api.thistoken.ai/register
---
Ready to try it yourself? Sign up at https://api.thistoken.ai/register to get your API key and start building.
Bạn muốn thử Token.AI?
Tạo API Key cấp dự án, bật kênh trong bảng điều khiển và định cấu hình định tuyến, ngân sách và nhật ký kiểm tra.
注册 ThisToken.AI 并获取 API Key