How I Fixed My Mess of an Issue Tracker with AI: A Post-Mortem and Playbook

1. First, How I Screwed It Up

Last year, the issue tracker of one of my small open-source projects accumulated over two hundred issues. As an independent developer, my approach to handling the issue tracker went through three stages—each one a textbook example of what not to do.

Stage 1: Reading everything manually. I spent an hour a day going through issues. I couldn't remember what I read, and even when I did, I couldn't connect the dots. The bug user A reported and the bug user B reported were actually the same problem—I only realized it after fixing it twice.

Stage 2: Bulk-closing issues. When I got busy, I batch-closed stale issues, and ended up closing a few genuinely valuable feature suggestions—later, a competitor shipped them, and only then did I go back and see that one of those issues had over twenty 👍.

Stage 3: Hook up AI directly, one-click summary. Sounds right, but I made a subtler mistake: I dumped entire pages of raw issues into a general-purpose model and asked it to "summarize this for me." What came back was a pile of correct-sounding fluff like "users reported several issues that deserve attention"—every word was accurate, but not a single line told me what to do next.

The biggest lesson from this stage: AI isn't mind-reading. If you feed it garbage raw material, it hands you back processed garbage.

2. Where the Problem Really Was

Looking back, the pain points of handling issues for independent developers and small teams are actually very specific:

1. Low information density. In a given issue, the genuinely useful parts might be three lines of error logs and the reproduction steps; the rest is emotions, speculation, and "+1"s.
2. Signal drowned by noise. Duplicate reports, "fake bugs" caused by environment differences, and off-topic discussions all mixed together.
3. What's missing isn't summarization, it's distillation. I don't need "there are 30 bug reports"; I need "those 30 reports are actually 7 distinct problems, ranked by impact—the top two account for 12 and 9 reports respectively, with representative links."

The failed AI usage was essentially asking AI to do "compression"; what I needed was "distill + deduplicate + rank"—and these three things can only be achieved through a structured process and explicit instructions.

3. The Right Workflow

After adjusting, I settled on a fixed four-step process, run once a week, taking less than half an hour each time.

Step 1: Export raw data. Use GitHub CLI to pull all open issues, keeping titles, bodies, comment counts, emoji reaction counts, and labels. This step ensures the AI gets the full context, not fragments like screenshots of web pages.

Step 2: Clean and chunk. For large batches of issues that exceed the model's context, split them by label or time period. For comments, keep only the top few by upvotes—they often contain reproduction details and environment info.

Step 3: Structured distillation. Use the prompt template below (see Section 4) to force the AI to output in fixed fields, forbidding vague generalities.

Step 4: Human confirmation before action. For the representative entries in each issue cluster the AI outputs, I always click through to the original link and check manually. AI compresses two hundred issues down to ten; the judgment call on those final ten is still my job.

4. A Copy-Paste-Ready Prompt Template

你是一位资深开源项目维护者。我会给你一批GitHub issue的原始内容
（标题、正文、关键评论、点赞数、标签）。请完成以下提炼：

1. 归并：判断哪些issue描述的是同一个根因问题，合并为issue簇，
   每簇给出：簇名称、包含的issue编号、总互动量（评论+emoji）。
2. 分类：将每个簇标注为 [bug] / [功能建议] / [文档问题] / [无效或重复]。
3. 排序：按"影响面（簇内issue数量）× 严重程度"给出优先级，
   说明排序理由，一句话即可。
4. 行动建议：对前3个簇各给出一条具体的下一步动作
   （如"补一个复现脚本"、"确认是否为某版本回归"）。
5. 信号提取：列出被多人点赞但未被归入主要簇的建议，
   这些可能是被低估的需求。

要求：
- 每个结论必须引用具体issue编号，禁止无出处的概括。
- 不确定是否同根因的，宁可单独成簇，并标注"待人工确认"。
- 输出使用markdown表格+简短说明，总长度控制在原文的10%以内。

The key design elements of this template are: mandatory citation of issue numbers (prevents hallucination), allowing uncertainty (prevents forced merging), and bounding the output length (prevents verbosity).

5. Before and After Using AI

The most striking change: I used to feel the issue tracker was "such a mess there's nowhere to start"; now it's "this week, tackle cluster one and cluster two." The psychological difference is even bigger than the time difference.

6. A Few Additional Notes

• Model choice: Tasks like clustering and classification demand strong long-context and comprehension capabilities, so I recommend a model with reasoning ability. As for API costs, check the official pricing page—at a once-a-week cadence, the cost is usually negligible for an independent developer.
• Don't chase full automation: I tried letting AI automatically label issues and post replies—the error rate was unacceptable. AI does the distillation, humans make the judgment calls. Don't reverse this division of labor.
• Accumulation beats one-off output: Save your weekly distillation results. A month later, you'll have an evolution trail of user demands—which problems keep recurring, which suggestions are gaining traction.

Final Thoughts

The issue tracker is the place closest to your real users, but for small teams it's also the easiest place to lose control. AI can't fix bugs for you, but it can turn two hundred pieces of noise into ten decisions waiting to be made—and "making decisions" is precisely the part maintainers should never outsource.

If you want to get this workflow running, you'll need a stable, transparently priced model API service—check out https://api.thistoken.ai/register . Sign up and start immediately, and with the template above you can run your first issue distillation session.

---

Tired of juggling provider integrations? Register at https://api.thistoken.ai/register and call every model through one base_url.