Failing First: How We Finally Got AI to Write Our Test Case Change Reports

Wait—the title should match the content provided. Let me output properly:

I. Start with the Failures: How We Got This Wrong

We're a five-person team responsible for maintaining a core business system. After each weekly iteration, a batch of test cases gets added, modified, or deprecated, and the test lead needs to compile a "Test Case Change Report" to share with development and product.

This sounds simple, but we tried three approaches in sequence, and all of them failed.

Failure #1: Dumping the entire case library export into the AI.

At first, we imagined it beautifully: throw the test platform's export file (thousands of rows of Excel) wholesale at a large language model and let it "summarize this week's changes." The result: the model either truncated the content due to excessive length or started fabricating with a straight face—the report contained case IDs that didn't exist at all. Verifying the report once took the test lead more time than writing it by hand.

Failure #2: Letting the AI run automatically every day and accumulate a weekly report.

The second time we got smarter and switched to daily increments. But we never defined what counts as a "change," so the AI counted formatting adjustments and owner renames as "case changes." By the end of the week, 90% of the report was noise, and the truly important case modifications were drowned in it—nobody read it.

Failure #3: The report was generated, but nobody dared to send it directly.

The third version's quality was finally acceptable, but the team developed a bad habit—copy-pasting the output straight into the group chat. One time, the AI categorized a "deprecated" case as "optimized," a developer made code changes based on that misunderstanding, and two days of rework followed. The problem wasn't the AI; it was that we skipped human confirmation and never required the AI to output traceable evidence.

Three failures taught us: Whether AI can help you depends on what you feed it, what rules you make it output by, and whether you keep a human in the review loop.

II. The Right Path: A Fixed Pipeline

After the retrospective, we redesigned the process around three steps: narrow the input → structured generation → human sign-off.

Step 1: Narrow the Input—Feed It Only the Delta

We wrote a dozen-line script that runs every Monday at dawn, compares Git versions of the case library (we migrated test cases to Markdown + Git management), extracts this week's diff lines, and attaches case IDs, modules, and authors. The output stays within one to two hundred lines, far below the model's context pressure line. This step solved the "fabrication" problem—the AI can only see real changes.

Step 2: Structured Generation with a Fixed Prompt Template

We polished a prompt template that constrains three key things: change classification criteria, output format, and mandatory citation of original case IDs. Here's the template—feel free to copy and use it:

你是一名测试团队的报告助理。我会提供本周测试用例库的Git差异记录。

你的任务：
1. 将变更分为四类：新增用例、逻辑修改、描述/格式调整、废弃用例。
   注意：仅改动措辞、负责人、标签的，归入"描述/格式调整"，不算逻辑修改。
2. 对"新增"和"逻辑修改"两类，各挑选影响最大的5条，用一句话说明其业务影响。
3. 输出格式：
   ## 本周变更概览
   - 新增：X条 | 逻辑修改：X条 | 格式调整：X条 | 废弃：X条
   ## 重点变更说明（附用例编号）
   ## 建议开发关注点
4. 每条说明必须附带对应的用例编号，禁止编造未在输入中出现的编号或内容。
5. 如果某条变更你无法判断属于哪一类，单独列入"待人工确认"清单，不要猜测。

以下是差异记录：
{diff_content}

Step 3: Human Sign-Off—The AI Only Drafts

The generated report goes to the test lead first, who focuses on reviewing the "Key Change Explanations" and "Suggested Focus Points for Development" sections—this is where the AI's judgment carries the most weight, and where a human must serve as gatekeeper. Only after confirmation does the report go to the team group. The entire review process takes about ten minutes on average.

III. Before and After: Let the Data Speak

One bonus: because case IDs are mandatorily written into the report, developers can now jump directly to the corresponding cases to check details, shortening the communication chain.

IV. A Few Practical Tips

1. AI's value lies in "organizing and summarizing," not in "detecting changes." Leave change detection to diff tools; let AI turn mechanical deltas into human-readable narratives—each does what it's best at.
2. Classification criteria in the prompt matter far more than writing style requirements. Our biggest improvement came from explicitly defining "what doesn't count as a change," not from making the report "sound more professional."
3. Mandatory ID citation is a low-tech anti-hallucination measure, but it works. Unsupported claims can't be written, and with citations, humans can easily verify.
4. On cost, tasks at this weekly-report scale consume very little usage. Refer to the official pricing pages for specific model billing—for us, the cost is negligible.

If you also want to hand this kind of "has to be done every week, nobody wants to do it" documentation work to AI, this pipeline is a good place to start. We currently call multiple models through a unified API gateway to run these tasks, which makes integration and key management much easier. If you're interested, check it out: https://api.thistoken.ai/register

---

Tired of juggling provider integrations? Register at https://api.thistoken.ai/register and call every model through one base_url.