## The Business Pain Point: Smart Search Looks Simple, bu...

The Business Pain Point: Smart Search Looks Simple, but Managing It Can Spiral Out of Control

I lead a small three-person team, and our main business is building mini-programs and tool sites for small and medium-sized merchants. Starting last year, clients began asking for "smart search" features: an e-commerce client wanted "search for red dresses but not waist-cinched ones," a content site client wanted semantic retrieval to replace keyword matching, and a local services client wanted users to describe their needs in a single sentence.

Each of these requirements is easy on its own—hook up a vector model for Embedding, connect an LLM for query rewriting, and you can get a demo running in a week. The real headache is the operations after deployment:

First, models are scattered across four vendors. One for Embedding, another for query rewriting, and a third vendor's small model for intent classification. Each vendor means one account, one billing system, one SDK, one rate-limiting scheme. When a client's service throws errors at midnight, I first have to figure out whether it's a vendor API outage or a bug in our code.

Second, clients must not affect each other. If client A runs a promotion and traffic spikes, without isolation, clients B and C will see slower or failed searches. But having a small team maintain a separate call pipeline for every client triples the maintenance cost.

Third, models can't be locked in. The rewriting model that performed best last quarter may be crushed by a new model this quarter, or its price may go up. If you hardcode vendors in your code, every model switch means a full regression test—something a small team can't afford.

As the person managing the process, my judgment is: the problem isn't the algorithms, it's governance. What we lack isn't model capability, but a unified control layer.

Architecture Design: Moving "Model Calls" Out of Business Code

The refactored architecture has four layers:

业务层：各客户的搜索入口（小程序 / H5 / API）
   ↓ 统一请求格式
网关层：统一 AI API 网关（多模型路由 / 限流 / 预算 / 日志）
   ↓ 按路由规则分发
模型层：Embedding 模型 / 查询改写模型 / 意图分类模型（可替换）
   ↓
检索层：向量库 + 倒排索引混合检索

There are three core principles:

Business code only knows "tasks," not "vendors." The business layer sends out an "embedding task" or "rewriting task"; which model it actually routes to is determined by the gateway's configuration.
Quota isolation per client. Each client gets its own key or namespace, with rate limiting and budgets enforced at the gateway layer, so client A's traffic spikes never drag down client B.
Every call leaves an audit trail. Who, when, which model was called, and how many tokens it cost are all recorded uniformly, making cost allocation and incident retrospection easy.

Why a Unified Gateway Reduces Maintenance Costs

This is the part of the solution I most want to emphasize. From a manager's ledger perspective, costs drop in four places:

First, integration cost drops from N times to 1. The gateway exposes a unified interface upward, and business code is written once. After that, no matter how many vendors you add or models you swap, business code needs zero changes—switching models goes from "full regression" to "change one line of routing config." For a small team, this frees up your most expensive resource—engineer time—from repetitive integration work.

Second, the troubleshooting path gets shorter. Before, when something went wrong, we had to hop between multiple vendors' consoles checking statuses; now all call logs, error codes, and latency data live in one place. With unified monitoring, fault localization goes from "guessing" to "looking."

Third, risk gets a brake. Budget controls at the gateway layer let you set hard limits per client. If a client's API gets maliciously hammered, it burns their own quota, not the whole account. Keys are also centrally held on the gateway side—no vendor keys are stored on business servers, drastically shrinking the attack surface.

Fourth, you gain leverage in vendor negotiations. Once call volume is concentrated in one place, model selection is fully decoupled—use whichever vendor offers the best price-performance, with switching costs approaching zero.

Key Implementation Steps

We rolled this out over four weeks, with the following key actions:

第 1 周：梳理与冻结
  □ 盘点所有 AI 调用点（改写/Embedding/意图分类），明确输入输出契约
  □ 定义统一的任务描述格式与错误码规范
  □ 在网关注册各模型，用真实流量样本做基准确认效果无损

第 2 周：灰度迁移
  □ 新旧链路并行，10% 流量切到网关
  □ 对比延迟、Token 消耗、检索命中率，连续 3 天达标后逐步放大到 100%
  □ 旧直连代码保留一周作为回滚通道

第 3 周：治理配置
  □ 按客户配置独立 Key、限流阈值、月度预算上限
  □ 配置告警：错误率 > 2%、延迟 P95 > 3s、预算用量 > 80% 时通知
  □ 建立变更流程：换模型必须先在灰度客户验证 48 小时

第 4 周：收尾固化
  □ 下线旧直连代码与散落的供应商密钥
  □ 输出一页运维手册：常见错误码含义、降级预案、模型切换 SOP
  □ 每月复盘各客户的 Token 成本与搜索质量指标

The part most worth highlighting is the fallback plan: the gateway has a model fallback chain configured—if the primary rewriting model times out three times, it automatically switches to a backup small model. The quality dips slightly, but search never goes down. In the three months since launch, we've been through two vendor hiccups with zero client-side impact.

Results and a Few Management Takeaways

The most visible change after migration: model-related maintenance hours dropped from about 6 hours per week to under 1; onboarding a new client's smart search requirement went from two weeks to three days, because the entire call pipeline and governance configuration are reused.

If you're leading a small team doing similar work, here are my three takeaways:

Governance first, intelligence second. Poor search quality can be tuned gradually, but an out-of-control call pipeline burns money and loses clients outright. Build the gate first, then add models.
Canary releases aren't process burden—they're rollback insurance. Route every model change through canary releases. Small teams don't have dedicated QA, so canary deployment is the lowest-cost quality defense.
Leave headroom in client commitments. Only with a fallback model chain and budget caps in place can you dare to write availability commitments into contracts.

The algorithmic barrier to smart search is dropping fast. What really separates the winners is who can manage model calls both reliably and cheaply. If you're also planning to consolidate your model calls, you can start by registering a trial on a unified AI gateway and migrating one of your existing call pipelines over for a week—the cost and stability math will become clear on its own: https://api.thistoken.ai/register

---

Every example in this post runs with a single API key — get yours at https://api.thistoken.ai/register and start in minutes.