## Foreword

Foreword

We're a five-person team building a vertical interest community. Three months after launch, daily comment volume grew from a few hundred to nearly five thousand, and problems followed: with only one person on operations, spam ads, flame-bait, and genuine user feedback started mixing together in the comment section. Valuable voices were drowned out, and risky content slipped through review. As the team lead, I have to admit we let this drag on for two weeks—not because it was technically hard, but because I hadn't figured out "who owns this process."

This article doesn't discuss model principles. It's written from a manager's perspective: how we designed an automated comment classification and labeling solution, how we drew the line between human and AI responsibilities, and how we controlled costs and risks. It's for people on similarly small teams who want AI to genuinely enter their business workflows.

Business Pain Points: Three Questions to Answer First

Before getting started, I forced myself to answer three questions—management homework I'd recommend to any team before introducing AI:

First, who decides the classification criteria? The boundaries of comment content are an operations judgment, not a technical one. The line between "complaining about the product" and "malicious attacks" is for operations to call. So the first step wasn't picking a model—it was sitting down with operations to define the taxonomy. We eventually converged on six categories: normal discussion, product feedback, violation risk, spam ads, flame-bait, and pending manual judgment. Each category came with three to five examples and handling rules.

Second, who catches it when AI gets it wrong? AI classification will never be 100% accurate. Our principle: AI only does initial screening and labeling, never final judgment. High-risk actions (deleting comments, banning accounts) must be confirmed by a human before execution; low-risk actions (folding ads) can be automated, but everything must be logged for traceability.

Third, how do we manage costs? Calling a large model for every single comment means the bill spirals out of control at scale. This drove our tiered design (see below) and was also the direct reason we later adopted a unified AI API gateway.

Architecture Design: A Three-Layer Pipeline

The overall architecture is a three-layer pipeline:

评论进入
   │
   ▼
┌─────────────┐
│ 第一层：规则引擎 │ ── 命中黑名单关键词/明显广告特征 → 直接打标，零成本
└─────────────┘
   │ 未命中
   ▼
┌─────────────┐
│ 第二层：轻量模型 │ ── 小模型快速分类，置信度 ≥ 0.9 → 采纳结果
└─────────────┘
   │ 置信度低 / 高风险类
   ▼
┌─────────────┐
│ 第三层：大模型复审 │ ── 强模型 + 详细 few-shot 提示词 → 输出结构化标签 + 理由
└─────────────┘
   │
   ▼
低风险 → 自动处置并记录
高风险 → 进入人工审核队列（附带AI理由，供审核员参考）

The value of this tiered design is managerial: roughly 70% of comments are intercepted by the first two layers, and less than 30% reach the large model, keeping costs under control. Meanwhile, everything entering the manual queue comes with the AI's classification rationale, so reviewers aren't judging from scratch—they're just "confirming or overturning," which is far more efficient.

Key Implementation Steps

Step 1: Define the label schema. Use JSON Schema to constrain model output and eliminate parsing issues from free-form text:

{
  "category": "enum: [normal, feedback, risk, spam, flame, unsure]",
  "confidence": "0-1",
  "reason": "一句话分类依据",
  "suggested_action": "enum: [pass, fold, queue_manual]"
}

Step 2: Write the classification prompt. The prompt includes definitions for each of the six categories, edge cases (provided by operations), and output format requirements. Edge cases are especially important—"this app is really hard to use" is feedback, while "whoever made this app has something wrong with their brain" is flame-bait. After writing examples like these into the prompt, the misclassification rate dropped noticeably.

Step 3: Build the pipeline. A message queue consumes comment events, filters them layer by layer, and writes results into the database with labels applied. We built a minimal internal page for the manual review queue, where every reviewer "confirm/overturn" action is recorded. This data gets fed back once a week to optimize prompts and add edge cases, forming a closed loop.

Step 4: Roll out gradually. We started with labeling only—no actions—for five days, with manual spot-checks comparing against AI results. Only after the agreement rate met our threshold did we enable low-risk automated actions like auto-folding. High-risk actions always retain a human confirmation step.

Why a Unified AI API Gateway Reduces Maintenance Costs

This deserves its own section, because it was a choice made after stepping on landmines. Initially we connected directly to one vendor's model API and quickly ran into three problems. First, we wanted different models for different tasks (a small model for lightweight classification, a strong model for re-review), but every vendor's SDK, authentication, and parameter formats were different, leaving our code full of branches. Second, one vendor rate-limited us, the pipeline backed up, and the comment section descended into chaos for another two days. Third, when we wanted unified statistics on call volumes and costs across stages, we found there was no ready-made data.

After switching to a unified AI API gateway, all these problems were solved at once:

• One interface, multiple models. The classification logic stays the same; switching models means changing a single model-name parameter. When we ran A/B comparisons between lightweight models, we finished in half a day without touching any business code.
• Multi-source redundancy with automatic failover. The gateway has backup model sources configured. If the primary source gets rate-limited or times out, it switches automatically, so the pipeline no longer stalls due to a single vendor's outage.
• Unified metering and budgeting. All calls go through the same gateway, so call volumes, success rates, and costs are transparent. I can keep tabs on spending with a weekly report—no more asking developers to pull data.

For a small team, this is essentially outsourcing "model operations" to a specialized layer, while the team itself focuses on classification logic and business workflows. The maintenance time saved is worth far more than the gateway's usage cost.

Three Takeaways from a Manager's Perspective

1. AI adoption is first about process design, then technology selection. Once the classification criteria, the human-AI responsibility boundaries, and the fallback mechanisms are settled, the technical implementation is actually not that hard.
2. Don't skip gradual rollout and traceability. AI making mistakes isn't scary—what's scary is nobody knowing it made them. Logs, spot-checks, and a human-in-the-loop are prerequisites for daring to automate.
3. Abstract away the model layer. Models will change and vendors will go down; your business logic shouldn't swing along with them.

Two months after launching this solution, the manual review workload for the comment section dropped to about 30% of what it was, while missed-content complaints actually decreased—because machines don't miss things due to fatigue. If you're building a similar AI application and want to experience multi-model switching and a unified gateway at low cost, check out this platform: https://api.thistoken.ai/register

---

Every example in this post runs with a single API key — get yours at https://api.thistoken.ai/register and start in minutes.