A Counterexample to Start

When many indie developers and small teams receive a request to "build a customer service bot," their first instinct is: organize the FAQ docs into Q&A pairs, stuff them into a vector database, hook up a large model for retrieval augmentation, and call it done.

This solution looks amazing during the demo phase: a customer asks "how do I return an item," and the bot answers instantly; ask "how much is shipping," and it responds fluently. So the team launches with full confidence.

Three months later, the numbers look ugly:

• When a user asks "why hasn't the order I placed last week arrived yet," the bot returns the standard "logistics turnaround time" answer, because the retrieval only matched the keywords "order" and "logistics";
• When the user says "that's not what I asked," the bot starts repeating the same answer;
• All the truly high-value inquiries—complaints, refund disputes, multi-condition combined queries—get handed off to human agents, who complain that "the bot takes all the easy questions and leaves me all the hard ones."

The root of the problem isn't that the model isn't strong enough—it's that the architecture only has "single-turn Q&A" with no "conversation state." Single-turn FAQ is a retrieval problem; multi-turn conversation is a state management problem. The two require completely different technical approaches.

Common Failure Patterns First

Before laying out the right path, let's list these pitfalls clearly:

Failure pattern one: brute-forcing with a bigger model. After switching to a flagship model, the small-talk experience does improve, but when a user follows up with "so what about my order," the model still doesn't know which order is being asked about—because there's no order information in the context at all. Costs multiplied severalfold, and the problem remains unsolved.

Failure pattern two: stuffing all historical messages verbatim into the prompt. On the surface it's "multi-turn," but in reality every call carries dozens of history messages. Token costs skyrocket, and irrelevant history dilutes the key information, so answer quality actually drops.

Failure pattern three: writing a separate set of calling code for every channel. One set for the WeChat official account, one for the website widget, one for the in-app experience—each handling model calls, timeout retries, and key management on its own. Three channels means three sets of keys, three sets of retry logic, three log formats. Changing one feature means changing it in three places. Small teams simply can't afford this level of maintenance.

Failure pattern four: using the same model for the entire pipeline. Intent recognition, answer generation, and summarization/compression are tasks with vastly different levels of difficulty. Using a flagship model for everything results in a monthly bill higher than a human agent's salary.

The Right Path: A Layered Architecture

Split the bot into four layers, each with its own responsibility:

用户消息
   │
   ▼
┌─────────────┐
│ 意图识别层    │ ← 小模型，判断：闲聊/FAQ/业务办理/投诉
└─────────────┘
   │
   ├─ 闲聊 → 轻量模型直接回复
   │
   ├─ FAQ → 检索增强（RAG）：向量库召回 + 重排 → 生成答案
   │
   └─ 业务/多轮 → 进入对话状态机 ↓
        ┌──────────────────┐
        │ 对话管理层          │
        │ - 会话状态槽位       │
        │   (订单号/日期/诉求) │
        │ - 缺槽追问          │
        │ - 澄清歧义          │
        └──────────────────┘
              │
              ▼
        工具调用：查订单API、发起退款等
              │
              ▼
        统一AI API网关 → 按任务路由到不同模型

The core idea: single-turn problems go through retrieval, multi-turn problems go through a state machine, and tool calls solve the problem of "the bot doesn't know the facts."

Key Implementation Steps

Step 1: Layer your intents—don't let every message go to the big model. Use a cheap small model (or rules + embedding similarity) to triage first. In practice, over 70% of messages are FAQ and small talk, which don't need an expensive model in the full pipeline at all.

Step 2: Design conversation state slots. Using the "check logistics" scenario as an example:

SLOTS = {
    "query_logistics": {
        "required": ["order_id"],
        "optional": ["time_range"],
        "ask_rules": {
            "order_id": "请问是哪个订单呢？可以提供订单号或下单手机尾号。",
        }
    },
    "refund": {
        "required": ["order_id", "reason"],
        "ask_rules": {
            "reason": "方便说下退款原因吗？未收到货、质量问题还是其他？"
        }
    }
}

def process_turn(state, user_msg, llm):
    # 1. 用LLM抽取本条消息中的槽位值
    extracted = llm.extract_slots(user_msg, SLOTS[state.intent].keys())
    state.slots.update(extracted)
    # 2. 检查必填槽位，缺了就追问，不硬答
    for slot in SLOTS[state.intent]["required"]:
        if not state.slots.get(slot):
            return SLOTS[state.intent]["ask_rules"][slot], state
    # 3. 槽位齐了，调工具拿事实，再生成答案
    facts = call_tool(state.intent, state.slots)
    return llm.generate_answer(user_msg, facts, state.history_summary), state

The key to this pseudocode is step 2: when information is missing, ask—don't force a generic answer. This is the dividing line between an FAQ bot and a conversational bot.

Step 3: Compress history instead of carrying it in full. After each turn, use a small model to compress the conversation into a one-sentence summary stored in state. What gets passed to the next turn is only the summary + slots + the last two turns of raw text. Token consumption drops by an order of magnitude, and key information isn't lost.

Step 4: Set human handoff rules. If clarification fails twice in a row, the user's sentiment turns negative, or there's a monetary dispute, hand off to a human agent immediately with the full conversation summary attached. The bot isn't meant to replace humans—it's meant to filter work for humans.

Why a Unified AI API Gateway Reduces Maintenance Costs

This is the piece small teams most easily overlook, yet it saves the most effort.

Recall failure pattern three mentioned earlier: writing separate calling code for each channel. Even with just one channel, if you use model A for intent recognition, model B for answer generation, and model C for summarization, you're managing three SDKs, three sets of keys, three error code formats, and three billing statements.

The value of a unified AI API gateway:

• One codebase to access all models. The gateway provides a unified interface spec—switching or adding models is just changing a model name parameter, without touching business code. A/B testing a new model becomes a ten-minute task.
• Centralized key and rate-limit management. Key leakage risk is consolidated to one place, and quota management no longer requires switching between five dashboards.
• A natural observation and cost control point. All calls go through the same gateway, so token consumption by intent and by channel is clear at a glance. Which layer should be downgraded to a cheaper model, which layer is being abused—let the data speak.
• Failure fallback. If an upstream model times out, the gateway layer automatically fails over to a backup model, invisible to end users.

For small teams, this is effectively outsourcing "platform engineering" to the gateway, so you can focus purely on business logic.

Suggested Rollout Rhythm

Don't aim for perfection in one shot: in week one, launch only FAQ retrieval to validate recall quality; in week two, add intent routing and slot follow-up questions; in week three, integrate tool calls—keeping a rollback switch at every step. A customer service bot is operated into shape, not built in one development cycle.

If you're about to get started, get the model-calling layer working first—register a unified AI API gateway account, obtain multi-model calling capability in a few minutes, and start validating your first multi-turn conversation scenario from an FAQ prototype: https://api.thistoken.ai/register

---

Tired of juggling provider integrations? Register at https://api.thistoken.ai/register and call every model through one base_url.