## The Business Pain Point

The Business Pain Point

A friend runs a five-person team doing cross-border e-commerce, selling home goods with stores on three platforms. Customer service is their biggest cost headache:

• About 2,000 inquiries per day, 70% of which are repetitive questions: shipping tracking, return/exchange policies, size inquiries
• Of the remaining 30%, many are "semi-complex" questions: order address changes, bundle discount calculations, initial complaint assessment
• Genuinely complex disputes requiring human agents account for less than 5%, but human agents are drowning in the first two categories and have no bandwidth for real work

They previously tried a "one-size-fits-all" approach: routing everything to a cheap small model. The error rate for address changes and discount calculations was too high, and customer complaints actually increased. Switching to an expensive flagship model burned over 20,000 yuan per month in Token bills, and the boss cried uncle.

The essence of the problem isn't "which model to choose," but not allocating model resources according to problem difficulty.

Architecture Design: Three-Tier Routing + Unified AI API Gateway

Core idea: route customer service requests by difficulty into three processing paths, managed through a unified gateway that handles all model calls.

User message
   │
   ▼
┌─────────────┐
│ Intent classifier │ ← lightweight model, classification only, negligible per-message cost
└─────────────┘
   │
   ├── L1 Simple inquiries (~70%) ──→ FAQ library + vector retrieval, small model generates reply
   │                              Cost: nearly zero, response <2 seconds
   │
   ├── L2 Semi-complex (~25%) ──→ Mid-tier model + order API tool calls
   │                              Cost: medium, response 3-5 seconds
   │
   └── L3 Complex disputes (~5%) ──→ Flagship model analysis + ticket draft generation → escalate to human
                                  Humans only confirm and communicate

Why does this save money? The key is the change in cost structure:

L1 uses a small model plus caching, driving the marginal cost of repetitive questions toward zero; L3 is only 5% of traffic, so even using the most expensive model doesn't hurt. Expensive models go where they matter most, not sprinkled everywhere.

Key Implementation Steps

Step 1: Set Up the Unified AI API Gateway (Days 1-2)

This is the foundation of the entire architecture. The gateway centrally manages all model calls—the benefits are detailed below.

Step 2: Build the Intent Classification Layer (Days 3-5)

The classifier doesn't need to be perfect—85% accuracy is good enough. The fallback strategy for misclassifications: automatically escalate to L2 when the L1 path fails to resolve the issue for two consecutive turns.

The core structure of the classification prompt:

CLASSIFY_PROMPT = """你是电商客服意图分类器。
将用户消息分为三类，只输出类别编号：

1 = 简单咨询（物流、退换政策、产品参数、尺码）
2 = 半复杂（改地址、优惠计算、发票、多订单关联）
3 = 复杂纠纷（投诉、赔偿、平台介入、情绪激烈）

判断依据：
- 涉及写操作或计算 → 至少 2
- 出现投诉关键词或负面情绪 → 3
- 拿不准时宁可高判，不要低判

用户消息：{message}
类别："""

Step 3: Implement the Three Paths (Days 6-12)

L1 path checklist:

□ Vectorize user message → retrieve Top 3 from FAQ library
□ Similarity > 0.82 → small model generates reply based on retrieved results
□ Similarity < 0.82 → downgrade to L2 handling
□ Log the session; unresolved after 2 consecutive turns → escalate to human
□ Sample 1 in 50 for manual review; error cases flow back into the FAQ library

The L2 path requires attaching tools to the mid-tier model: order query API, address modification API, and promotion rule calculator. The model only handles intent understanding and tool invocation—calculations go to deterministic code. Never let an LLM do arithmetic.

The point of the L3 path isn't to replace human replies, but to let the flagship model do the information organization upfront: order timeline, summary of customer demands, suggested resolution. When a human agent receives the ticket, they see the conclusions directly—average case handling time dropped from 15 minutes to 6 minutes.

Step 4: Monitoring and Iteration (Ongoing)

After launch, review three things weekly: whether routing ratios are drifting, resolution rates per path, and the distribution of reasons for human escalation.

Why a Unified AI API Gateway Reduces Maintenance Costs

This is the point I most want to emphasize. Before the overhaul, their problem was: three platforms, four or five models, each with its own SDK and Key—just "changing one model's configuration" meant touching code in three places.

With a unified gateway:

1. One Key, one endpoint. Application code talks only to the gateway. Switching models (e.g., changing L2 from model A to model B) requires only a gateway config change—zero changes to business code. After launch, we did one model swap that took 10 minutes; previously it would have taken at least half a day.

1. Unified billing and usage observability. Costs across the three paths are clear at a glance. Insights like "L2 accounts for 60% of the bill but only resolves 25% of problems" are simply invisible without a unified entry point.

1. Unified rate limiting, retry, and fallback logic. Write it once, shared by all paths. One day when the small model provider had issues, the gateway automatically failed over to a backup model—business teams never noticed.

1. Keys are no longer scattered across the codebase. After that scare where a "departing employee took configuration files with them," this is a security baseline for small teams.

By rough calculation, the gateway reduced this five-person team's model-related maintenance time from about 10 hours per week to 2 hours—for indie developers, the time saved is the most expensive cost of all.

Conclusion

The entire overhaul took three weeks, with a total investment of less than one person-month. In return, monthly costs dropped from over 20,000 yuan to under 5,000, and three support staff were freed up for product selection and operations. For indie developers who want to move fast, my advice is: build the gateway first, then the routing, and optimize individual paths last—get the order wrong and you'll rework endlessly.

If you want to try this architecture hands-on, start by signing up for a unified AI API gateway service and get multi-model management working first: https://api.thistoken.ai/register

---

Ready to try it yourself? Sign up at https://api.thistoken.ai/register to get your API key and start building.