Common Failure Patterns First

Last year I took over operations for a medium-sized website whose Nginx access logs ran several GB per day. My boss's request was simple: "The API feels sluggish lately—maybe scrapers are leeching it? Go take a look."

My first thought: we have AI now, so why not just throw the logs at it and let it analyze them?

That's how I ended up with the following "cautionary tales."

Anti-pattern #1: Upload the entire log file. I packaged up the day's access.log and sent it to an LLM—it immediately blew past the context limit and never even got through the door. Even with a model supporting ultra-long contexts, tens of thousands of highly repetitive log lines contain maybe a few dozen genuinely anomalous entries. The model's attention gets diluted by the flood of normal requests, and it outputs nonsense like "detected a large number of requests with 200 status codes."

Anti-pattern #2: Only take the first hundred lines. Scraper behavior is precisely long-cycle and low-frequency. An IP making one request every 30 seconds looks "completely harmless" in any isolated segment. Truncating to the head of the file throws away the most valuable dimension: time.

Anti-pattern #3: Ask AI to directly "identify who the scrapers are." Without decision criteria, the model can only guess based on UA strings. Any halfway decent scraper disguises itself with a normal browser UA these days, so the conclusions you get are basically fortune-telling.

Anti-pattern #4: Have AI write one-off analysis code and discard it after running. The next day when logs refresh, you have to re-describe the requirements from scratch. After a few rounds of this, the whole workflow gets left in a corner and abandoned.

The common thread in these four pitfalls: treating AI as a "black box that reads logs" rather than a "collaborator that helps you complete the analysis." Once I figured that out, I switched to a different approach.

The Right Path: Humans Aggregate, AI Judges

The core idea is one sentence: first use a one-line command to aggregate logs into "behavior profiles," then have the AI perform anomaly assessment on those profiles.

Raw logs are too primitive, but once aggregated into "a behavior summary per IP," the data drops from millions of lines to a few hundred—right in the sweet spot of what LLMs handle best, and with extremely high information density.

The process has three steps:

Step 1: Aggregate. Use awk or have AI write an aggregation script that summarizes the logs per IP across these dimensions:

• Request count, unique URL count, unique session count
• Average request interval and its standard deviation
• Distribution characteristics of accessed paths (whether it traverses listing pages, whether it scrapes in ID order)
• Number of UA types, whether UA changes over time
• Status code distribution (lots of 403/429 usually means it's been blocked already but keeps pushing through)
• Whether access times concentrate in early morning hours

Step 2: Assess. Hand the aggregated results along with business context to the LLM, and have it do three things: identify suspicious IPs with justifications, classify them by suspicion level, and recommend corresponding actions.

Step 3: Institutionalize. Have the AI distill this round's reasoning into a reusable rules document. Next time, just re-run the aggregation and apply the same prompt—results in five minutes.

A Copy-Paste Prompt Template

你是一名网站安全分析助手。我会提供一份按 IP 聚合的访问日志行为摘要
（CSV 格式），字段包括：ip、请求数、独立URL数、平均请求间隔（秒）、
间隔标准差、UA种类数、UA变化、状态码403/429次数、访问时段分布、
是否按ID顺序抓取、是否遍历列表页。

背景信息：
- 网站类型：{填写你的网站类型}
- 正常用户典型行为：{例如：通过搜索进入，浏览3-10个页面，停留几分钟}
- 已知合法爬虫：{例如：Googlebot、Bingbot}

请你完成：
1. 标记可疑 IP，按「高/中/低」三级分类；
2. 每个可疑 IP 必须给出具体依据（引用字段数值），不允许只给结论；
3. 区分「恶意爬虫」「过激的合规爬虫」「疑似脚本化的正常用户」；
4. 对每一类给出处置建议（限流、封禁、加入robots协议等）；
5. 指出摘要数据中你无法判断、需要我补充原始日志佐证的部分；
6. 最后输出一份可直接保存的判定规则文档，供下次分析复用。

数据如下：
{粘贴聚合后的CSV}

The key design elements of this template are items 2 and 5: forcing the model to provide evidence, and forcing it to acknowledge its judgment boundaries. The former prevents hallucinated accusations (embarrassing if you block a CDN node thinking it's a scraper); the latter prevents blind trust in its conclusions.

Before and After AI

The most practical change: previously, going from "the API is slow" to pinpointing a specific malicious IP meant days of digging through logs; now one pass through the workflow yields a tiered list and handling recommendations the same day. What remains is manually verifying raw log snippets for the highly suspicious items—AI narrows the scope, humans make the final verdict. This division of labor is far more reliable than "leave it all to AI."

One more note on cost: LLM APIs bill by token, and aggregated data typically amounts to only tens of thousands of tokens per call. Check the official pricing page for exact figures, but for a small team it's usually negligible—the real barrier was never money, but methodology.

A Few Lessons Learned

1. AI's value is in assessment, not storage. Don't expect it to remember your million-line logs; feeding it "one profile line per IP" is the correct approach.
2. Always require the model to give evidence. "IP x.x.x.x is suspicious" is worthless; "IP x.x.x.x has an average interval of 0.4 seconds with a standard deviation near zero, and its UA changed 5 times within an hour" is valuable.
3. Make AI admit uncertainty. Explicitly ask the model to list the parts requiring human verification—it greatly reduces the risk of blocking legitimate users.
4. Solidify the process into documentation. A successful analysis that isn't distilled into a reproducible workflow might as well never have happened.

If you don't have a go-to model API yet, check out this platform—pair your aggregated data with the prompt template, and you can run your first scraper analysis right after signing up: https://api.thistoken.ai/register

---

Tired of juggling provider integrations? Register at https://api.thistoken.ai/register and call every model through one base_url.