## A Common Failure Scenario

A Common Failure Scenario

Last year, when I took over a legacy project that had been running for four years, the first thing I did sounded reasonable: I threw the entire repository at AI and asked it to "map out this project's feature evolution timeline."

The result was a disaster.

The AI returned a summary that looked neat at first glance: it listed some commits chronologically, gave each commit a概括性的 headline, and paraphrased the feature list from the README. The problem was—it hadn't mapped an "evolution," just a "change ledger." Which features were iterations of the core storyline? Which version was a turning point in direction? Refactoring and feature additions were mixed together, with no discernible rhythm. Worse, it sorted commits from merged branches by author date, completely scrambling the timeline.

Another colleague on the team tried a different approach: only letting the AI read the release notes. This time it was even simpler—those release notes were updated only once every six months, so the resulting timeline was miles away from the actual code evolution.

These two failures made me realize something: it's not that AI is incapable—the problem is that we treated "mapping the evolution timeline" as a one-shot generation task. It's actually an analysis task that requires structured input and staged reasoning.

Three Common Traits of Failed Approaches

Looking back, common mistakes basically fall into three categories:

One: Too much data fed in, too little structure provided. Paste thousands of commit messages directly, and the AI's attention gets diluted by irrelevant details, so it can only write about surface-level things. What's truly valuable in Git history is the "patterns of change," not individual records.

Two: Treating raw output as conclusions. The output of git log --oneline contains tons of noise—merges, chores, typo fixes. Feed it in without cleaning, and the AI's summary will naturally be noise too.

Three: Asking for all the answers at once. An instruction like "give me a complete evolution report" forces the AI to make things up without intermediate reasoning. Break it into several steps, and the quality will be completely different.

The Right Path: The Three-Step Feeding Method

The adjusted workflow looks like this:

Step 1: Clean and structure the data. First, process the Git history locally into a format the AI can easily digest. The core command:

git log --no-merges --pretty=format:"%ad|%s" --date=short \
  | grep -v -E "^(.*\|)?(chore|typo|fix typo|bump)" > git_history.txt

This removes merge commits, filters out noise, and outputs in a date|commit message format. If you're organizing by milestones, you can add --grep to extract sections by tag or keyword, saving each section separately.

Step 2: Ask questions in stages. First, have the AI cluster commits by time period and identify the feature themes of each stage; then have it compare adjacent stages to find "turning points"—for example, when a category of commits drops from 30% to nearly zero, it usually means the feature was replaced or abandoned. These patterns are hard to spot by manually browsing logs.

Step 3: Make it verify itself. For every conclusion, require the AI to cite the commit dates and entries it's based on. Any conclusion without supporting evidence must be labeled "speculation." This step blocks most hallucinations.

A Reusable Prompt Template

你是一位资深技术分析师。我会提供一份项目的Git提交历史,
格式为「日期|提交说明」,已过滤merge和杂项提交。

请完成以下分析:

1. 将提交按功能主题聚类(如:认证模块、支付流程、性能优化),
   每个主题列出涉及的提交数量和时间跨度;
2. 按季度划分阶段,概括每个阶段的重心,并指出重心如何转移;
3. 标记3-5个关键转折点(新模块首次出现、某类提交突然增多
   或消失),每个转折点引用具体日期和提交说明作为依据;
4. 输出一份时间线表格:阶段 | 时间范围 | 核心变更 | 依据提交数。

要求:
- 所有结论必须能追溯到提供的提交记录,无法追溯的标注[推测];
- 不要美化或脑补未体现的意图;
- 如果数据太短或太稀疏,直接说明无法分析,不要硬编。

Before and After

Before using AI (manual browsing): Mapping the evolution of that four-year-old project took me three working days on and off, and the resulting timeline document still missed two important architectural shifts—because they were hidden among dozens of inconspicuous rename commits. When doing project handover for new colleagues, I wasn't even fully confident in what I was presenting.

After using AI (three-step method): With data cleaning plus three rounds of questioning, I got an evidence-backed evolution timeline within half a day. The AI found two turning points I had originally missed: one where cache-layer commits appeared intensively in a certain month and then vanished completely (later confirmed as the scheme being overturned), and one where the proportion of test-related commits doubled (corresponding to the team getting serious about building test infrastructure). The handover document went from "me talking from memory" to "every conclusion traceable by date."

Time isn't the most critical factor. What matters most is that the trust structure of the output changed: manual summaries rely on memory and impressions, while staged AI analysis produces outputs with built-in evidence chains that can be reviewed and corrected.

A Few Additional Tips

• For repositories with poor commit message quality, have the AI categorize first, then analyze. For a history full of "fix" and "update" style messages, you can first have the AI assist clustering based on file path changes, which carries more information than plain text.
• Process large repositories in segments. Split the history by year or tag, analyze in batches, then have the AI merge the results—much better than feeding everything at once.
• Manually spot-check conclusions. Pick 5-10 key nodes and verify them in the repository. This step cannot be skipped—the AI's traceability annotations themselves can be wrong.

If you're looking for model resources suited for this kind of structured analysis task, check out Thistoken.ai. It provides unified access to multiple models, making it convenient to switch models based on task characteristics for workflows like history analysis. See the official pricing page for details: https://api.thistoken.ai/register

---

Tired of juggling provider integrations? Register at https://api.thistoken.ai/register and call every model through one base_url.