Let's Be Honest: Our First Approach Was Completely Wrong

We're a five-person team maintaining a fast-iterating business system. Every week, test cases get added, removed, and modified: new regression cases added, outdated cases deprecated, parameterized data adjusted. The boss and clients both want to know "what exactly changed in our test assets this week," so the job of producing a "weekly test case change report" fell to the QA team.

Our first automation attempt was a Python script that diffed two weeks of test case files and output the raw differences. The result was a pile of JSON diff fragments that nobody but the script's author could understand. The second attempt was "smarter": we dumped the entire diff into a large language model and asked it to "write a report for us." The AI did write one—flowing prose of two thousand words—but half of it was fabricated. It invented "deletion reasons" for removed cases that never existed, and made up "impact module" correlation analyses. That report got cited by a client to question requirement changes, and we were embarrassed for a week.

Looking back, the reasons for failure are clear:

1. We mistook "generation" for "understanding": diff output is a machine's view—it's meaningless to humans directly, and feeding it to an AI just makes the model guess the context.
2. We gave the AI no boundaries: if you don't tell it "don't make up information that doesn't exist," it will absolutely fill the gaps with fluent language—that's the nature of large language models.
3. One-shot generation, no verification: every "why it changed" statement in the report should have a source, but we had no traceability mechanism at all.

What Is the User's Pain Point, Really

Setting aside our own detours, this problem hurts every small team the same way:

• Time-consuming: manually compiling a weekly report—digging through records, reconciling versions, writing explanations—eats up half a day;
• Inconsistent quality: when things get busy, the report becomes a perfunctory list of events that nobody reads or trusts;
• Knowledge lives in individual heads: whoever wrote the cases remembers the reasons; as soon as they take leave, the report stops coming;
• Format changes every time: recipients (boss, clients, QA leads) each want different things, leading to constant rework.

What AI can do is precisely not "write it for you," but translate structured diff data into contextualized, well-formatted, trustworthy narratives—provided you set up the process correctly.

The Right Process: Three Steps

Step 1: Let Machines Do Machine Work

Test cases live in a Git repository (this matters—it gives you version history for free). A scheduled script runs every Friday and produces a structured diff: which case IDs, titles, and modules were added; which were deleted; which fields were modified. At the same time, export the week's commit messages and requirement ticket numbers—this is the authoritative source for "reasons for change," and it's the only material the AI is allowed to draw from.

Step 2: Let AI Do AI Work

Feed the structured diff + commit notes + a fixed report template to the model, and let it do the translation and categorization: group by module, distill common causes, flag risk levels, generate a summary. The key constraints go into the prompt: no speculation, no adding information that doesn't exist, every conclusion must map to a record in the input.

Here's the prompt template we refined after learning the hard way—feel free to copy and adapt it:

你是一名测试团队的报告撰写助手。请根据我提供的【结构化diff】和【commit说明】,
生成一份周度测试用例变更报告。

规则(必须遵守):
1. 只使用输入中出现的信息,禁止推测或编造变更原因;
2. 每条变更必须能对应到输入中的用例ID或commit编号;
3. 若某条变更没有对应说明,在"原因"栏写"待补充",不要自行填写;
4. 输出格式:
   ## 本周测试用例变更摘要(3句以内)
   ## 按模块分组的变更明细(表格:用例ID|操作|标题|原因|关联需求)
   ## 风险提示(仅当输入包含回归/失败记录时才写,否则省略此节)
5. 语气客观简洁,不使用营销化措辞。

【结构化diff】:
{paste_diff_here}

【commit说明】:
{paste_commits_here}

Step 3: Let Humans Do Human Work

The QA person only does two things: spot-check two or three conclusions in the AI report to verify they trace back to the original records (one minute); and fill in the "to be added" reason fields. Within ten minutes, a trustworthy weekly report is out the door.

On cost: API token consumption depends on the scale of your team's test cases—check the official pricing page of the platform you use for exact costs. For a once-a-week batch task, it's usually not a burden.

Before and After AI

The biggest change isn't really saving time—it's that the report went from a "personal artifact" to a "team asset." Anyone taking over can understand the process, historical reports share a uniform format, and they can be rolled up directly into monthly and quarterly summaries.

Three Lessons for Small Teams

1. Fix your data before bringing in AI: get test cases into Git and attach explanations to changes. Skip this step, and the AI can only make things up.
2. The core of a prompt is "prohibitions," not "instructions": telling the model what it must not do does more for quality than telling it what to do.
3. Always keep a human spot-check step: AI handles mass production, humans provide the endorsement—don't reverse that division of labor.

If you also want to hand this kind of "once-a-week organizational work" to an AI pipeline and need a stable API channel, check out this platform: https://api.thistoken.ai/register —sign up and start using it right away. Hook it into your existing scheduled script, and your first auto-generated weekly report can go out next week.

---

Tired of juggling provider integrations? Register at https://api.thistoken.ai/register and call every model through one base_url.