## On the Day It Broke, Nobody Did Anything Wrong

On the Day It Broke, Nobody Did Anything Wrong

Friday afternoon, the most reliable developer on the team submitted a seemingly harmless PR: a minor version upgrade of a core dependency library. CI was all green, code review passed, merged and deployed.

At 8 PM, the alarm went off. That library had quietly removed a deprecated method in a patch release, and our code had a call path not covered in the test environment that happened to use it.

In the post-mortem, the problem was clear: the CHANGELOG had documented it all along — nobody just read it.

This wasn't an attitude problem. A mid-sized project easily has dozens of direct dependencies, and with transitive dependencies included, a single upgrade can involve hundreds of change entries. Asking engineers to read each one and judge "will this affect us?" is both tedious and inefficient. So everyone defaults to "minor versions should be safe," substituting guesswork for verification.

As a manager, I realized the real risk here wasn't a single incident, but the fact that "dependency change review" simply didn't exist as a process — it was so costly that nobody wanted to do it, so the risk was silently accepted.

AI Happens to Be Great at Exactly This Kind of Work

After that incident, I kept thinking: reading CHANGELOGs and assessing impact is fundamentally a task with "high information volume but clear judgment rules." Isn't that exactly what AI is good at?

Our approach was simple: feed the change log together with our usage patterns to AI, and let it do the first round of screening for us. The specific workflow:

Step 1: Gather materials. Before upgrading, have the engineer export two things: the CHANGELOG (or Release Notes) of the target version, and a list of actual calls to that dependency in our codebase. The latter can be generated with a simple grep or a dependency analysis tool — it doesn't need to be precise.

Step 2: Have AI screen for breaking changes. Give both materials to the AI together, using a fixed prompt template, and ask it to output a structured risk report.

Step 3: Humans only review the report. The AI flags entries like "removed APIs," "behavior changes," and "deprecation warnings," and checks each one against our call sites. Engineers only need to verify the items flagged in red, rather than reading through the entire log.

Step 4: Archive the report in the PR. The risk report is attached to the upgrade PR, giving reviewers something concrete to rely on and leaving a trail for post-incident accountability.

After running this process for a quarter, the most direct change: dependency upgrades went from "gut feeling" to "documented decisions." New hires can also follow the process to independently complete upgrade reviews, no longer relying on the implicit experience of senior staff.

The Prompt Template We Use

The template below can be copied directly — just replace the placeholders with your actual content:

你是一位资深的依赖升级审查助手。请严格按以下步骤分析:

【输入材料】
1. 依赖变更日志(原样粘贴):
<在此粘贴CHANGELOG或Release Notes>

2. 我们项目对该依赖的实际调用情况(方法/类/配置项清单):
<在此粘贴调用清单>

【任务要求】
1. 从变更日志中提取所有可能的破坏性变更,包括:
   - 移除或重命名的API、方法、参数
   - 默认行为的变更(默认值、返回结构、错误处理)
   - 废弃警告与配置项变更
   - 最低运行环境要求变化(Node/Python/JDK版本等)

2. 将每条破坏性变更与调用清单比对,标注影响等级:
   - 🔴 高:我们代码中存在直接调用,升级后必然报错或行为异常
   - 🟡 中:间接影响或边界场景,需人工确认
   - 🟢 低:与我们的使用方式无关

3. 输出格式:
   ## 风险摘要
   ## 🔴 必须处理的变更(附我们代码中的对应调用点)
   ## 🟡 建议人工复核的变更
   ## 🟢 无影响的变更(仅列条目名)
   ## 建议的升级路径(是否需要分版本逐步升级)

注意:不确定的地方明确标注"需人工确认",不要给出猜测性结论。

That last line in the template — "if uncertain, say so" — matters a lot. What we want is a reliable first-pass screening, not confidently wrong conclusions.

Before vs. After Using AI

Before:

Dependency upgrade reviews relied on experience; reviewers basically only looked at version numbers and diffs
CHANGELOGs were "for when there's time" — in practice, never read
Compatibility-breaking changes were only discovered after incidents, with average diagnosis times measured in hours
Upgrade risk was entirely borne by individual judgment, nobody wanted to touch major version upgrades, and technical debt piled up

After:

Every upgrade PR comes with an AI-generated risk report, giving reviews a concrete basis
Over 80% of breaking changes are identified before merging (this is our own team's experience after improving the process, not a universal statistic)
Incident diagnosis went from "digging through code" to "checking the report" — confirmed in minutes
Engineers are now willing to do version upgrades, because the cost of review has dropped to an acceptable level

From a manager's perspective, the biggest gain isn't time saved, but turning a "gray zone nobody owned" into a process node with deliverables and traceability.

A Few Implementation Tips

If your team wants to adopt this approach, here are some lessons learned:

Start with core dependencies. No need to cover all dependencies at once — pick the most frequently used ones first and build process momentum.
Maintain the call list. It's the foundation for AI judgment; if the list is outdated, the report becomes unreliable. You can generate it automatically on a schedule in CI.
Match model choice to task complexity. A lightweight model is enough for simple patch upgrades; major version migrations involve semantic judgment and warrant a stronger model — of course, for specific choices and pricing, refer to the official pricing page.
Persist the reports. Don't let risk reports live only in chat windows. Archive them in PRs or a knowledge base — they become audit assets for the team.

Final Thoughts

Dependency management, viewed as a single event, is trivial — but accumulated over time, it becomes systemic risk for the team. AI's role here isn't to replace engineers, but to turn review work that was "too expensive for anyone to do" into a step that fits naturally into the workflow.

If your team also manages model calls through a unified gateway and wants to institutionalize this kind of "AI reads docs and does first-pass screening" workflow into daily collaboration, check out ThisToken.AI — you can register here: https://api.thistoken.ai/register

---

Every example in this post runs with a single API key — get yours at https://api.thistoken.ai/register and start in minutes.