Background: A "Good Enough" Model Is Quietly Eating Your Profits
When many small teams build internal Q&A chatbots for their companies, the approach is simple: pick the most capable model and throw every question at it. The logic is "the chatbot is a core feature, so we can't cut corners."
That's exactly what our team did at first. For the first two months after our internal knowledge base Q&A went live, all requests went through the same high-end model. It wasn't until our monthly review, when we pulled the log data, that we discovered an awkward fact:
About 78% of questions were high-frequency, simple ones like "How do I file an expense report?", "What's the WiFi password?", or "Where can I check my remaining annual leave?" The answer quality for these questions was virtually indistinguishable between the most expensive model and a mid-tier model—but the difference in the bill was very real.
Even more painful was the time cost: every day, about 40+ questions needed manual reprocessing due to queuing, timeouts, or context length limits, with ops colleagues spending an average of 30-40 minutes per day on them. This time doesn't show up on the API bill, but it's real labor cost.
Core Idea: Split by Scenario, Not by "Gut Feeling"
We eventually split the chatbot's requests into three scenarios, each with a different model strategy. The split wasn't based on model leaderboards, but on question complexity and cost sensitivity.
Scenario 1: High-Frequency Standard Q&A (about 70-80% of traffic)
Typical questions: policy lookups, process guidance, common troubleshooting.
Characteristics: answers can almost always be found in knowledge base documents, questions are phrased relatively consistently, and reasoning requirements are low.
Model strategy: a lightweight, fast-responding model + a smaller max_tokens limit.
We set both the thinking budget and output length very conservatively for this tier. After the switch, the per-request cost for this segment dropped to roughly one quarter of the original, while the "inaccurate answer" rate in user satisfaction surveys stayed unchanged.
Scenario 2: Multi-Document Synthesis and Long-Context Understanding (about 15-25% of traffic)
Typical question: "What are the standards for cross-department business trip approval and expense reimbursement respectively? Please put them into a comparison table."
These questions require the model to read multiple documents and synthesize the information, placing real demands on long-context capability and instruction following.
Model strategy: a mid-to-high-end model, paired with a larger context window and a more refined system prompt.
We didn't cut costs here. This proved to be right: in early attempts to force lightweight models to handle these questions, a single incoherent answer would trigger two or three rounds of follow-ups from users, resulting in higher total token consumption and a worse experience.
Scenario 3: Fallback and Human Handoff (about 5% of traffic)
Typical questions: vague inquiries, questions outside the knowledge base's scope, clearly emotional complaints.
Model strategy: the most capable model for intent recognition and fallback answers; when recognition fails, generate a ticket summary and hand off to a human.
This segment is small, but it determines users' "baseline impression" of the entire chatbot—it's worth using the best model.
Before vs. After the Split
| Dimension | Before (single high-end model) | After (per-scenario models) |
|---|---|---|
| Scenario coverage | 100% of traffic through one model | Dedicated strategy per scenario |
| Cost per simple Q&A | Baseline 100% | ~25% |
| Average first-token latency for simple questions | Baseline | Significantly shorter (lightweight models infer faster) |
| Daily manual interventions | 40+ | ~10 |
| Daily ops handling time | ~30-40 minutes | ~10 minutes |
| Overall monthly API bill | Baseline 100% | ~40% |
| Answer quality for complex questions | Good | Equal or better (prompts tuned per scenario) |
Back-of-the-envelope math: API costs dropped by about 60%, ops saves about half an hour per day, which adds up to roughly half a person-day per month that can go into feature iteration. For a team of two or three people, this "invisible income" is worth more than the bill itself.
Key Infrastructure: Why You Need a Unified Gateway
The plan sounds great, but there's a practical issue in implementation: the three scenarios may involve models from two or three different providers. If each scenario connects directly to each provider's API, you'll run into—
- Three sets of auth, retry, and logging code: rewriting the glue logic every time you add a model;
- Inconsistent API formats: switching models means changing the request body structure, with high regression testing costs;
- No fallback when a model is discontinued or rate-limited: if a provider's model suddenly goes down, the corresponding scenario collapses.
Our approach was to route all requests through a single gateway layer. The benefits show up most directly in three things:
- Switching models is a one-line config change. Scenario 1 routes to model A; one day model A raises prices or degrades, and you switch to model B by changing a model name—zero code changes. Once, due to rate limiting on the provider's side, we switched Scenario 1's model within ten minutes with no user-visible impact.
- Unified billing and usage dashboard. Cost, latency, and error rates for all three scenarios are compared in a single panel. Our monthly "model strategy review" takes half an hour instead of exporting three providers' bills and stitching them together by hand.
- Independent rate-limiting and degradation policies per scenario. Scenario 1 has high volume but high fault tolerance, so it can degrade aggressively; Scenario 3 is the fallback, so it gets the highest priority. This kind of fine-grained control is nearly impossible with direct connections.
The essence of the gateway's value is: it downgrades "which model to choose" from an architectural decision to a configuration item. With the model market changing this fast, this flexibility is survival-level for small teams.
Recommended Rollout Path
If you want to replicate this approach, follow this order for minimal risk:
- First, pull two weeks of historical question logs and do a simple scenario distribution analysis—your distribution will likely differ from our 75/20/5;
- Switch the largest scenario to a lightweight model first and observe answer quality metrics for a week; this segment offers the biggest gains at the lowest risk;
- Handle the fallback scenario last, and polish the human-handoff ticket summary workflow.
The whole process doesn't require rewriting your chatbot, provided you already have (or can easily integrate) a unified gateway that supports multi-model routing.
If you haven't built this layer of infrastructure yet, check out the one we're currently using: sign up and start using it right away, with unified multi-model access and per-scenario routing, config changes taking effect in real time—well suited for teams that want to validate the "per-scenario model switching" playbook at low cost first → https://api.thistoken.ai/register
---
Ready to try it yourself? Sign up at https://api.thistoken.ai/register to get your API key and start building.
Token.AI を試してみませんか?
プロジェクトレベルの API Key を作成し、コンソールでチャネルを有効にして、ルーティング、予算、監査ログを設定しましょう。
注册 ThisToken.AI 并获取 API Key