Why You Need This Article
Last week I was running a private beta of a small tool that needed to automatically classify and tag user questions. My initial workflow went like this: sign up with one model provider, read the docs, apply for a Key, write code, run it once, feel like the results weren't accurate enough; then sign up with another provider and repeat the whole process. Three days later, I had accumulated five console accounts and a pile of Keys whose purposes I couldn't remember—and in the end, I still picked a model based on a "gut feeling."
Then I changed my approach: I wrote a local script that dispatches the same test set to multiple models, scores them uniformly, and outputs a comparison table. The entire process was compressed into a single afternoon. From three days of gut feeling to three hours of data—that's what this article gives you.
For indie developers and small teams, the hidden costs of the model selection phase are often underestimated: switching back and forth between consoles, adapting to each vendor's SDK, manually comparing outputs—add it all up, and this time often exceeds the actual development time. Unified access + scripted A/B comparison is the first step toward turning this process into a product.
Preparation: One Key, Multiple Models
ThisToken.AI provides a unified API gateway compatible with the OpenAI interface format. This means:
- You only need to register one account and manage one Key;
- To switch between different models under the same
base_url, you only need to change themodelparameter; - Your A/B script doesn't need a separate adapter for each model vendor.
The registration process is simple:
- Open the ThisToken.AI website and register an account;
- Go to the console, create a new Key on the API Key page, and store it safely;
- Confirm that the models you want to compare are available in your account (the available model list in the console is authoritative);
- For costs, refer to the pricing page on the official website—we make no numeric promises here.
Your First Working Code
The Python script below does something very simple: it defines a small test set, calls two models in turn, and prints the results side by side. Once you get it running, you've completed the minimal closed loop of multi-model A/B testing.
import os
import time
from openai import OpenAI
client = OpenAI(
api_key=os.environ["THISTOKEN_API_KEY"],
base_url="https://api.thistoken.ai/v1",
)
# 你的测试集:真实场景中建议准备 20~50 条有代表性的输入
test_cases = [
"帮我判断这条用户反馈是投诉还是咨询:'付款之后一直没收到确认邮件'",
"把这句话归类为技术问题或产品问题:'App 在弱网下会闪退'",
"提取这句话里的产品名和情绪倾向:'你们这个新版本太难用了'",
]
models = ["gpt-4o-mini", "deepseek-chat"] # 换成你账号里可用的模型
def ask(model: str, prompt: str) -> tuple[str, float]:
start = time.perf_counter()
resp = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
temperature=0, # A/B 测试时固定随机性,保证可复现
)
elapsed = time.perf_counter() - start
return resp.choices[0].message.content.strip(), elapsed
for model in models:
print(f"\n===== 模型: {model} =====")
for i, case in enumerate(test_cases, 1):
answer, elapsed = ask(model, case)
print(f"\n[用例 {i}] 耗时 {elapsed:.2f}s\n{answer}")Before running, install the dependencies and set your Key:
pip install openai
export THISTOKEN_API_KEY="你的Key"
python ab_test.pyThree Upgrades: From "It Runs" to "It's Useful"
The above is the minimal version. To make it truly support selection decisions, I recommend doing three things:
First, write output to files instead of the terminal. Write each model's results into CSV or Markdown, along with latency and token usage (the response object has a usage field). When you're comparing three to five models, visually comparing terminal output quickly becomes unmanageable.
Second, fix the temperature and run multiple times to take the average. Single-response latency fluctuates a lot; run each test case three times and take the average latency—only then will your conclusions hold up.
Third, add a layer of automatic scoring (optional). If your task has objective criteria (such as classification labels), you can call another model as a judge to compare outputs against the ground truth. The entire pipeline then becomes: request → collect → score → aggregate, with no need to watch the screen manually.
A practical data point: for my own classification task test set—30 cases, 3 models—the full script run takes just a few minutes. Before that, using the "manually copy-paste and try one by one" approach, just organizing the various console accounts took me half a day. What you save isn't only waiting time, but the continuity of your attention—which is especially valuable for indie developers.
Common Pitfalls
- Don't hardcode your Key into the script. Use environment variables or a
.envfile, and double-check before committing the script to version control. - Don't be too aggressive with concurrency. If you want to use multithreading to speed things up, probe the rate limits with a small batch first, and implement backoff logic for retries on errors.
- Model names are authoritative per the console. Model naming varies significantly across vendors—before copying someone else's sample code, confirm that the model identifier is available.
- Estimate costs from the pricing page. Billing varies considerably across models and input/output volumes. Run a small-scale test before large test sets, and use the official pricing page for cost estimates.
Final Thoughts
Model selection shouldn't rely on impressions—it should rely on a comparison table you generate yourself. A local script, a unified gateway, and a test set of a few dozen entries can turn "choosing a model" from mysticism into engineering. If you haven't registered yet, you can start here: https://api.thistoken.ai/register —register, get your Key, and run the code above; one afternoon is more than enough.
---
Ready to try it yourself? Sign up at https://api.thistoken.ai/register to get your API key and start building.
Token.AI を試してみませんか?
プロジェクトレベルの API Key を作成し、コンソールでチャネルを有効にして、ルーティング、予算、監査ログを設定しましょう。
注册 ThisToken.AI 并获取 API Key