## Opening with a Counterexample

Opening with a Counterexample

I took over a legacy project that had been running for six years—over two hundred functions, zero unit test coverage. My first reaction: isn't this exactly the kind of work AI excels at?

So I opened a chat window, dumped the entire code directory into the model in one go, and typed a prompt worthy of being recorded in my personal hall of failures:

> "Generate unit tests for all functions in this project."

The model worked hard and spat out several hundred lines of test code. I eagerly ran pytest—the screen filled with red. Half the tests failed on import because the model guessed the module paths wrong; the other half failed all assertions because the model didn't truly understand what those undocumented, confusingly named legacy functions were actually doing. There was also a fair share of tests whose assertions reflected the behavior the model imagined the functions should have, rather than their actual behavior.

That night, I deleted all the generated tests. Coverage remained at zero.

Three Common Traits of the Failed Approach

Looking back, this kind of "dump everything at once" failure was almost inevitable, for very typical reasons:

First, greed. Asking AI to process dozens of functions at once exceeds the effective working range of the context, and quality degrades as the number of functions increases.

Second, no context provided. What legacy projects lack most is documentation. Function signatures are incomprehensible, parameter meanings have to be guessed, and implicit preconditions are known to no one—if you don't feed this information to the AI, it can only fill in the gaps with "generic imagination," and the tests it writes naturally won't match reality.

Third, vague requirements. The phrase "generate unit tests" doesn't specify the testing framework, whether the database should be mocked, or the priority of edge cases. The AI has to improvise, and the direction it improvises in is often not what you want.

In other words, the AI's capability for this task is sufficient—we just used it wrong.

The Right Path: Small Batches, With Context, Skeleton First, Flesh Later

After adjusting my strategy, I restructured the process into four steps, and the results were immediate.

Step one: have the AI do a "checkup" first, not "surgery." Pick three to five representative functions (one pure function, one with a database dependency, one depending on an external API), feed them to the AI along with the type definitions, constants, and upstream/downstream call code they use, and ask it to first output: how testable is this function, what does it depend on, and where should mocking start. The deliverable of this step is understanding, not code.

Step two: explicitly establish the conventions for the test skeleton. Which test framework to use, naming rules, mocking strategy, and which categories of cases each function must cover at minimum (happy path, boundary values, abnormal inputs)—all of it written into the prompt. This becomes the reusable "mold" for all subsequent batches.

Step three: loop in small batches. Process only three to five functions at a time—generate a batch, run it, fix a batch. The most common error in AI-written skeletons is mock object behavior that doesn't match the real dependency; in small batches, these issues take minutes to fix, whereas in large batches they pile up until you break down.

Step four: skeleton first, assertions later. The compromise I ultimately adopted: have the AI generate the complete test skeleton—test classes, test functions, case naming, input data construction, mock setup, all ready to go—while the key assertions are left for human confirmation, or for the AI to fill in based on "actual runtime output." This way, the AI's efficiency advantages (building structure, generating data, enumerating coverage of various scenarios) are fully leveraged, while what it's bad at (determining the true intent of legacy code) is vetted by a human.

A Reusable Prompt Template

This is the skeleton generation template I've settled on—just replace the bracketed content:

你是一个资深测试工程师。请为下面的函数生成单元测试骨架。

## 项目约定
- 测试框架：[pytest / JUnit5 / vitest 等]
- mock 库：[unittest.mock / Mockito 等]
- 测试文件命名：test_<模块名>.py，测试函数命名：test_<被测函数>_<场景>
- 遵循 Arrange-Act-Assert 结构

## 生成要求
1. 为每个函数至少生成：正常路径 1 个、边界值 2 个、异常输入 2 个用例
2. 所有外部依赖（数据库、网络、文件、时间）必须 mock，不许真实调用
3. 每个测试函数体内：Arrange 和 Act 部分写完整可运行的代码，
   Assert 部分先写 TODO 注释，标注“需要人工确认期望值”
4. 不要修改被测函数本身
5. 如果函数存在你无法确定的行为，在测试文件顶部用注释列出疑问清单

## 被测函数及其依赖代码
[粘贴函数代码]
[粘贴相关的类型定义、常量、被调用的下游函数签名]

The key items are #3 and #5: leaving assertions as TODOs forces you to actually look at the AI's list of open questions—which often contains the most dangerous implicit logic in a legacy project.

Before and After Using AI

Before (purely hand-written): Hand-writing tests for a medium-complexity function (one database dependency, two branches)—including reading the logic, setting up mocks, and debugging—averaged 25 to 40 minutes. For two hundred functions, it was unthinkable, which is why coverage stayed at zero for six years.

First attempt with AI (wrong approach): Generated 500 tests in one evening, spent an entire day debugging them the next day, then deleted them all. Net gain: negative. Only harvest: the lesson.

After (correct approach): In batches of 5 functions, AI generates the skeleton in a minute or two, and manually confirming assertions and fixing mocks averages 5 to 8 minutes per function. Over two weeks of spare time, test coverage of the core business modules went from zero to a maintainable level—and I understand what every single test in the repository is verifying.

The conclusion is clear: AI isn't an employee who writes tests for you—it's an assistant that carries away the heavy lifting of building skeletons and generating data. Get the division of labor right, and your efficiency doubles; get it wrong, and you're just practicing your typing speed while deleting code.

A Few Practical Tips

• Coverage is a false goal. Start with the modules that break most often, not by scanning files in order.
• Always run generated results locally before they enter the repository. AI has no execution environment, so there will always be code that "looks correct."
• If you're using an API for batch processing, bake the template into the system prompt and pass function code as the user message—token consumption becomes more controllable. For costs, refer to the official pricing page.
• The list of open questions is more valuable than the test code itself—it's an excavator for digging up the implicit logic of legacy projects.

If you're also paying down technical debt on a legacy project, or want your team to genuinely integrate AI into the daily development workflow instead of using it as a chat toy, give the Thistoken platform a try—it aggregates multiple model APIs, and you can start right after registering: https://api.thistoken.ai/register

---

Tired of juggling provider integrations? Register at https://api.thistoken.ai/register and call every model through one base_url.