First, the Common Failure Patterns

GitHub issues are one of the lowest signal-to-noise areas in open source projects and team collaboration. A single feature request might contain three paragraphs of environment description, two paragraphs of venting, and one paragraph of reproduction steps—with the actual request buried in the last sentence. So many indie developers and small teams think of AI: having a large language model read issues, distill information, and generate summaries sounds wonderful.

But in practice, the failure rate is surprisingly high. Based on my own missteps, the pitfalls fall into three categories:

Pitfall 1: Pasting the entire text without any trimming. Some people stuff the issue plus dozens of comments into the chat box and ask "summarize this for me." The model's output is often a summary that "covers everything but says nothing"—it treats venting and substance with equal weight, and after reading it you still don't know what the issue actually wants. The longer the context, the thinner the model's attention to key information spreads—this is a common flaw of large language models.

Pitfall 2: Instructions that are too vague. A prompt like "summarize this issue" hands the criteria for distillation entirely over to the model to guess. Once I asked AI to summarize an issue about a memory leak, and it returned a technical explainer—what memory leaks are and their common causes. The information wasn't wrong, but it was completely not what I wanted.

Pitfall 3: Letting AI give conclusions without requiring citations from the original text. When distilling long texts, AI engages in "plausible confabulation": summarizing users' speculations as facts, or merging the views of two different commenters into one person's. If the summary doesn't cite information sources, making decisions based on that distilled output carries considerable risk.

The common thread across all three pitfalls: treating AI as an "auto-summarize button" rather than an analysis tool that requires clearly defined task boundaries.

The Right Approach: Define Criteria First, Then Feed Materials, Finally Demand Evidence

After adjusting my approach, I restructured the process into three steps:

Step 1: Clarify the distillation goal. Before invoking AI, think through what you want to extract from the issue. The user's core request? Reproducible bug steps? The distribution of version compatibility issues? Different goals require completely different prompts.

Step 2: Preprocess the input material. I first use a simple script to separate the issue body from comments, label commenter identities (maintainer, reporter, bystander), and filter out noise comments like pure emojis and "+1"s. This step doesn't require complex code, but it significantly reduces the probability of AI being led astray by noise.

Step 3: Use a structured prompt requiring "conclusions + evidence." This is the key. My prompt template requires every conclusion the AI outputs to be accompanied by a citation from the original text, and to explicitly distinguish between "what the user explicitly said" and "what is inferred." This way, after receiving the results, I can quickly verify them and avoid being misled by AI confabulation.

A Reusable Prompt Template

Here's the template I use daily. You can copy it directly and replace the bracketed content:

你是一名技术项目助理，负责分析GitHub issue。请严格按照以下要求处理：

【输入材料】
Issue标题：[粘贴标题]
Issue正文：[粘贴正文]
相关评论：[粘贴评论，标注评论者身份]

【任务】
1. 提炼报告者的核心诉求（不超过3条），每条附带原文引用作为证据
2. 提取所有可复现步骤和关键环境信息（版本号、操作系统、依赖等）
3. 区分“用户明确陈述的事实”和“用户的猜测或推断”，分开列出
4. 如果存在多个评论者，分别标注各自的立场和关键信息
5. 指出信息缺口：如果要让维护者处理这个issue，还缺哪些必要信息

【输出格式】
- 核心诉求：（编号列表，每条带引用）
- 复现与环境：（结构化列表）
- 事实 vs 推断：（两个分区）
- 信息缺口：（列表）
- 一句话总结：（不超过50字）

【约束】
- 不得添加原文中没有的信息
- 引用必须是原文片段，不得改写
- 如果某项信息在原文中不存在，明确写“未提及”

The core design of this template is "every conclusion must have evidence, and every gap must be explicitly declared." In practice, the AI confabulation problem has essentially disappeared, because the constraint "citations must not be rewritten" limits the model's room for creative interpretation.

Before and After Using AI

Before using AI: Processing an issue with thirty-plus comments took me an average of 20 to 30 minutes of reading through, highlighting key points, and judging which information was trustworthy. For poorly worded reports, I had to flip back and forth through comments piecing together context. Handling a dozen issues a day meant massive time spent on "finding information" rather than "solving problems."

After using AI: With the preprocessing script plus templated prompting, the information distillation time per issue dropped to 3 to 5 minutes. More importantly, the output quality is consistent—because I require distinguishing facts from inferences and flagging information gaps, the summary I receive often lets me judge directly: does this issue lack information and need follow-up questions, or is the information complete enough to schedule? Once, from the AI's "information gaps" list, I noticed the reporter never mentioned their OS version—a single follow-up question resolved three days of back-and-forth.

To be clear, AI hasn't replaced my reading of the original text; it has replaced the "reading through and locating" step. Key decisions still require my verification of citations, but this already represents a qualitative leap in efficiency.

A Few Additional Suggestions

• Use different templates for different issue types. Bug reports, feature requests, and usage inquiries each have different distillation priorities—it's worth maintaining separate templates.
• Preprocessing is worth the investment. Even just filtering out noise comments and labeling commenter identities—five minutes of script writing yields far greater improvements in distillation quality than expected.
• Control input length. For extra-long discussion threads, distill in segments first and then aggregate; a two-stage approach works better than stuffing everything in at once.
• On cost, these tasks have moderate model capability requirements—mid-tier models are usually sufficient. For specific choices and pricing, refer to each platform's official pricing page.

If you're building your own AI workflow and need a stable API service with flexible switching across multiple models, check out Thistoken (https://api.thistoken.ai/register)—a unified gateway that makes it easy to quickly compare and switch models across different distillation tasks.

---

Tired of juggling provider integrations? Register at https://api.thistoken.ai/register and call every model through one base_url.