## First, How I Screwed It Up

First, How I Screwed It Up

Last year I added a simple feature to my side project: have AI analyze user behavior data. It sounded great—throw a CSV at a large language model, ask "how's my user retention," and wait for conclusions.

It crashed and burned, in a very typical way. I've summarized these failure patterns into three categories—you've probably seen them too:

Pitfall 1: Dumping raw data at the model. Stuffing tens of thousands of rows of raw logs into a conversation means the model either truncates it or starts making things up. The retention curve it gave me was suspiciously beautiful—when I checked against the raw data, I found it had mixed up registration dates and activity dates. But its tone was so confident I almost put it straight into my weekly report.

Pitfall 2: Expecting AI to deliver in one shot. "Analyze this data and give me growth advice"—prompts like this usually get you correct-sounding platitudes: "Consider improving user experience" or "You could try user segmentation." AI is great at writing conclusions that would hold true for any product.

Pitfall 3: Not verifying conclusions. AI said "Wednesday has the highest activity," I believed it, and adjusted my push notification schedule accordingly. Two weeks later I checked with my own SQL query—it was Sunday. The model had probabilistically picked an answer that "looked plausible."

The common thread in all three pitfalls: I treated AI like an analyst who understood my business, when it's really an executor that needs a strictly managed process.

The Right Path: Breaking AI into Four Stages of the Workflow

Once I figured this out, I restructured my process: AI doesn't produce conclusions directly—it handles the dirty work and framework work in the pipeline.

Step 1: AI describes and cleans the data, makes no judgments

I stopped asking "what does the data say" and only ask "what's in the data." Missing value distributions, field meanings, shapes of outliers—AI does this descriptive work quickly and reliably. Judgment stays with me.

Step 2: AI generates analysis code, humans review and execute

The key turning point: don't let the model "look at" the data—have it write code to process the data. Numbers from Python don't hallucinate, and I can review AI-written code line by line. If it's wrong, I fix the prompt and regenerate at nearly zero cost.

Step 3: AI interprets and challenges, rather than concludes

After the code runs, I put AI into "skeptic mode": Can this correlation be interpreted as causation? Is the sample size sufficient? Are there confounding variables? Its ability to nitpick far exceeds the self-review patience of a solo developer like me.

Step 4: AI translates conclusions into decision language

Only in the final step do I let it write things up—versions for different audiences: a to-do version for myself, an announcement version for users.

Before and After: What You Save Isn't Just Time

The most valuable change is the last one: once the process was in place, I was forced to first think clearly about "what do I want to know, and what would count as knowing it"—AI is just the accelerator. Since API calls are pay-as-you-go, one round of this process costs a controllable amount (check each model's official pricing page for specifics).

A Reusable Prompt Template

Step 2—generating analysis code—is the stage that works most smoothly for me. Here's the template, free to take:

你是一名数据分析工程师。我会提供一份数据集的字段说明，
请帮我生成一份可执行的 Python 分析脚本。

要求：
1. 只生成代码，不要直接给业务结论；
2. 代码包含：数据加载、缺失值/异常值检查、我提出的每个
   分析问题的对应计算逻辑；
3. 每段计算前用注释说明：这段在算什么、为什么这样算、
   结果如何解读（解读标准，不解读具体数值）；
4. 输出的每个关键数字都打印出来，方便我核对；
5. 如果某个分析问题在现有字段下无法回答，明确指出缺
   什么字段，不要硬算。

字段说明：
[在此粘贴字段名、类型、含义]

我的分析问题：
1. [问题一，例如：过去30天的新用户在第7天的回访率]
2. [问题二]

数据文件路径：[path]

For the Step 3 challenge stage, just replace item 5 in the template above with "请针对以下计算结果提出至少三个可能的解释偏差或方法论漏洞" (please propose at least three possible interpretation biases or methodological flaws for the following computation results).

Three Sentences for Indie Developers

1. AI's best position in data analysis is writing code, describing data, and playing devil's advocate—everything except "the person who gives conclusions."
2. Any specific number AI states must come from executable code, not the model's probabilistic output.
3. Prompts are the accumulated assets of your process—don't start from zero every conversation.

If you want to run this workflow in your own product and need a stable API access layer that can switch between multiple models, check out this platform—register and start using it right away: https://api.thistoken.ai/register

---

Tired of juggling provider integrations? Register at https://api.thistoken.ai/register and call every model through one base_url.