## 1

1. First, Let Me Tell You How I Botched This

After my small product launched last year, user feedback channels multiplied overnight: in-app forms, WeChat groups, email, and app store reviews. Within three weeks, I had accumulated over 500 items.

The first time I tried to organize them, I made several classic mistakes.

Mistake 1: Feeding all the raw feedback to AI in one go. I pasted all 500 messages to the model as-is and asked it to "summarize user needs." What came back was a pile of platitudes like "users want the product to be easier to use" and "some users have concerns about performance." The granularity got flattened completely, making it useless for prioritization.

Mistake 2: Feeding unclean data to AI. The WeChat group messages contained memes, small talk, "got it" replies, and screenshot paraphrases. The AI dutifully analyzed all of it, and the output included conclusions like "some users think the weather is nice."

Mistake 3: Letting AI act as both classifier and judge. I asked the model to directly output a "priority ranking" without giving it any criteria. The ranking it produced sounded convincing, but it was actually just ordering things by mention count—but are "slow startup" being complained about 80 times and "can't log in" being complained about 15 times the same thing? Obviously not.

Mistake 4: Trying to do it in one step, with no human verification. The AI lumped two contradictory pieces of feedback into the same category ("would like a dark mode added" and "dark mode is too harsh on the eyes"). I posted it to the team group without checking, and a colleague called it out on the spot. Quite embarrassing.

These four failures taught me: AI is not a wish-granting machine. When it comes to batch-organizing feedback, the failure wasn't AI's—it was mine, in how I used AI.

2. The Right Approach: A Four-Step Pipeline

After some painful reflection, I broke the process down into four stages, where AI handles only one task per step.

Step 1: Preprocessing and cleaning. Feed raw feedback to AI in batches (50-100 items per batch). The task is to remove small talk, strip out emojis and duplicates, and normalize colloquial phrasing—while not changing the original meaning—and to keep the original numbering for traceability.

Step 2: Structured categorization. Have AI output in a unified format: feedback ID, summary of the user's request, category (feature/performance/UX/Bug/other), affected feature module, and emotional intensity. The key here is to give it a closed list of categories and not allow it to invent new ones.

Step 3: Aggregation and weight calculation. This is the biggest difference from my earlier approach—priority isn't determined by AI "feel" but calculated by explicit rules. The rules I gave AI: mention frequency, impact scope (whether it blocks a core workflow), emotional intensity, and alignment with product goals—each dimension scored 1-5, then combined with weights. AI only scores according to the rules and explains its reasoning; the ranking is computed, and is reviewable and challengeable.

Step 4: Manual review of the output. I spend just 20 minutes checking two things: whether the Top 10 needs are correctly categorized, and whether any contradictory feedback was wrongly merged among the high-scoring items. Once confirmed, the final list is output.

3. A Reusable Prompt Template

Step 3 is the core. Here's the template I'm currently using:

你是一名产品分析师。我会提供一批已清洗的用户反馈(格式:编号|类别|诉求摘要|情绪强度1-5)。

请完成以下任务:
1. 将语义相同的诉求聚合为一个需求项,标注包含的反馈编号;
2. 对每个需求项按以下维度打分(1-5分):
   - 频次:提及该诉求的反馈数量(1条=1分,2-5条=2分,6-15条=3分,16-30条=4分,30条以上=5分)
   - 阻断性:是否导致用户无法完成核心流程(完全无法=5分)
   - 情绪:该组反馈的平均情绪强度
   - 战略匹配:是否符合本季度产品目标(我会在下方提供目标)
3. 加权总分 = 频次×0.2 + 阻断性×0.4 + 情绪×0.2 + 战略匹配×0.2
4. 按总分降序输出表格:需求项|聚合编号|各维度得分|总分|一句话理由
5. 单独列出"存在内部矛盾的需求"(同类反馈中出现相反诉求),不要合并它们。

约束:不要自创评分维度;每个得分必须给出依据;无法判断时标注"需人工确认"。

本季度产品目标:【填写你的目标】
用户反馈数据:【粘贴数据】

The table output from Step 4 can be pasted directly into Feishu/Notion, and then you just manually go through the items marked as "needs manual confirmation."

4. Before vs. After Using AI

Before AI (or more precisely, before using AI correctly):

• 500 feedback items piling up over three weeks, never daring to start
• Manual organization took about 6-8 hours per pass, and new feedback would come in halfway through
• Categorization standards changed every time, so last month's list couldn't be compared with this month's
• Prioritization was based on gut feeling, and in team discussions no one could convince anyone else

After the pipeline was running smoothly:

• For 500 feedback items, cleaning + categorization + scoring takes about 40 minutes, plus 20 minutes of manual review
• Categorization standards are fixed, run once a week, so lists can be compared over time
• Priorities have a computational basis; blocking bugs naturally rise to the top, and discussion efficiency improved noticeably
• The biggest gain was discovering several low-frequency but highly blocking issues—previously, when ranking by "who shouted loudest," they always sank to the bottom

It's worth noting that 5%-10% of AI's categorization output still needs manual correction, especially for sarcasm and screenshot paraphrases. What AI does is compress 6 hours of manual labor into under 1 hour—the judgment call remains in your hands.

5. One Final Note

This pipeline has certain requirements for the model: long context, stable structured output, and support for batch processing. I currently run this pipeline by calling multiple models through a unified gateway—cheap, fast models for cleaning, and more capable models for categorization and scoring. Choosing models by pipeline stage strikes a much better balance between cost and quality than "one model to rule them all." For specific pricing, refer to each provider's official pricing page.

If you'd like to build your own multi-model workflow, you can try ThisToken.AI's unified API: https://api.thistoken.ai/register — one base_url to switch between multiple models, with batch processing, retries, and cost tracking all in one place. It's a great fit for indie developers and small teams getting started.

---

Tired of juggling provider integrations? Register at https://api.thistoken.ai/register and call every model through one base_url.