rewritten = llm.chat(

model="gpt-4o-mini",

messages=[{"role": "user",

"content": f"改写为法律检索查询：{question}"}]

)

2. Vector retrieval + reranking, keeping only currently valid regulation versions

chunks = rerank(

vector_db.search(rewritten, top_k=20, filter={"valid": True})

)[:5]

3. Constrained generation: force cited sources

context = "\n---\n".join(c.text for c in chunks)

answer_text = llm.chat(

model="claude-sonnet",

messages=[{"role": "user", "content": f"""仅依据以下材料回答，

材料未涉及的部分明确说明无法回答，并标注引用条款：

材料：{context}

问题：{question}"""}],

timeout=30

)

return {"answer": answer_text, "sources": [c.meta for c in chunks]}

**Step 3: Build an evaluation set (Week 3).** We selected 120 real historical consulting questions and manually annotated the key points of the standard answers. Every time we changed the prompt or swapped models, we ran the set and let the scores speak—avoiding the illusion of "it feels better."

**Step 4: Gradual rollout (Week 4).** Started as internal assistance only: the AI produces a draft with citations, and a human reviews it before sending it to the client.

## IV. The Efficiency Math: Before and After

Comparison two months after launch (internal team statistics):

| Metric | Before launch | After launch |
|---|---|---|
| Average response time per inquiry | 48 hours | Draft within 20 minutes, same-day reply |
| Manual time per inquiry | ~50 minutes | ~12 minutes (verification only) |
| Daily processing capacity | ~35 inquiries | ~120 inquiries |
| Monthly AI call costs | — | ~1/8 the cost of adding one legal hire |

The same four people, more than triple the output—and with faster responses, client renewal rates also improved noticeably.

## V. Why We Insisted on a Unified AI API Gateway

LLM calls appear at four points in this pipeline: question rewriting uses a small model, generation uses a large model, and evaluation needs yet another model. In the first version, we put different vendors' keys in the configuration separately, and within two weeks we got burned three times: one vendor's API timed out with no automatic failover, taking the service down for half an hour; every model swap required code changes in three places plus a redeployment; and bills were scattered across four dashboards, making reconciliation take an entire afternoon.

After switching to a unified AI API gateway, the change in maintenance costs was tangible:

- **One key, one API protocol covering all models**. Switching models went from "change code + redeploy" to "change one parameter." During the model benchmarking exercise, we compared five models in one night with zero code changes.
- **Built-in retries, timeouts, and failover**. Automatic switching when upstream providers hiccup—that half-hour outage from before is now essentially imperceptible.
- **Call logs and usage statistics centralized in one place**. It's clear at a glance how much each stage costs, giving cost optimization a data foundation—for example, we discovered evaluation accounted for 30% of call volume, added a caching layer, and saved another chunk of money.

For a small team, building this kind of infrastructure in-house isn't realistic—plugging into an existing gateway is the most cost-effective choice. If you want to quickly get your own RAG or other AI application up and running, you can start by registering here: https://api.thistoken.ai/register

---

Ready to try it yourself? Sign up at https://api.thistoken.ai/register to get your API key and start building.