## First, the Failures

First, the Failures

Before taking on the analysis of a production service's access logs, I always thought "bot detection" was a solved problem: add a User-Agent blacklist, set up rate limiting, or at worst throw in a CAPTCHA—surely that would stop them.

Failure #1: the UA blacklist. Three days later, the logs filled up with utterly ordinary identifiers like Mozilla/5.0 (Windows NT 10.0), with request frequencies eighty times that of a human. The attacker's cost of switching UAs was zero; my cost of maintaining the blacklist was daily.

Failure #2: global rate limiting. I set "60 requests per minute per IP" and ended up hurting a batch of real users behind the company's egress NAT—customer support tickets arrived before any bot alert. Even more embarrassing, the bots used a pool of rotating low-frequency IPs, each sending only 20 requests per minute, dutifully slipping under my threshold.

Failure #3: dumping the logs straight into an AI with "help me find the bots." The model dutifully returned a textbook-style explainer of bot characteristics, along with a pile of generic suggestions—none of which applied to my millions of log lines. The problem wasn't the AI; it was me giving it a task equivalent to telling an intern "look at the data and share some thoughts"—no context, no criteria, no output format. Anyone would just phone in an answer.

The Right Approach: Treat AI as a Forensic Assistant That Reads Logs

I later came to a realization: the value of AI here isn't "knowing about bots"—it's that it can check, categorize, and produce evidence for patterns across millions of log lines that no human could ever read through, according to criteria you define.

There were three key shifts:

1. First, use the AI to help me clearly define "what counts as anomalous," rather than running the model first. I fed it the business context (normal user access paths, peak hours, typical session length), had it draft a checklist of anomaly hypotheses, and I manually confirmed and revised them.
2. Make the AI produce structured output, not free-form commentary. Every conclusion must come with evidence: which IP, which time window, what pattern triggered which hypothesis.
3. Closed-loop validation. For suspicious IPs flagged by the AI, sample and manually verify first, then decide whether to block. Solidified rules get reused in the next round.

The Concrete Workflow

Step 1: Preprocessing. The logs are too large to feed in whole. Slice them by hour first, then use a script to aggregate them into "IP-level session summaries"—for each IP: request count, unique path count, path entropy, UA count, request interval distribution, hit rate (200/404 ratio). This is work an ordinary script can do; don't waste tokens on it.

Step 2: Ask questions with hypotheses attached. Give the AI the summary samples together with business context, using this template:

Role: You are an access log forensic analyst.

Background:
- This is an [e-commerce/content/tool] website; normal users visit [N] pages per day and stay an average of [N] minutes
- Normal traffic peaks during [time window]; real users may share egress IPs
- The site has no public API; all machine requests should be treated as suspicious

Data: Below are hourly session summaries aggregated by IP, with fields including
[request count / unique path count / path entropy / UA count / request interval distribution / status code ratio]

Task:
1. Check each IP against the following anomaly hypotheses and determine which ones it hits:
   - Hypothesis 1: Extremely low variance in request intervals (machine rhythm)
   - Hypothesis 2: Path entropy per session is abnormally high or abnormally low
   - Hypothesis 3: Multiple UAs appear from the same IP
   - Hypothesis 4: High proportion of 404s / probing paths (.env, /admin)
   - Hypothesis 5: Highly consistent behavioral fingerprints across multiple IPs (distributed crawler)
2. What other hypotheses do you think are worth adding? Explain your reasoning
3. Output format: a table with columns [IP / hypotheses hit / evidence summary / confidence (high, medium, low)]
   Output only the table and additional hypotheses; do not write educational content

Step 3: Manual spot-checking. High-confidence results go straight to the watchlist; medium- and low-confidence ones get sampled against the raw logs. This step can't be skipped, but the workload changes from "reading millions of lines" to "reading the few dozen lines the AI circled."

Step 4: Rule solidification. Bake confirmed fingerprints (IP ranges, UA combinations, behavioral patterns) into the gateway layer, and record false positives as corrective input for the next round of prompts.

Before and After AI

The biggest change isn't "catching bots more accurately"—it's that every time I block, I can produce an explainable chain of evidence. Before, when I accidentally blocked real users, all I could do was apologize; now I can explain exactly why that IP was flagged and why it's been cleared this time.

A Few Reminders

• Don't throw raw logs in as-is. Aggregated summaries are more effective and cheaper than raw lines. Token costs depend on the model tier; refer to the official pricing page.
• Don't let the AI directly execute blocking actions. Its output should be "recommendations + evidence"—the authority to act stays in your rules system. Large models make confident mistakes, and for something like blocking IPs, the cost of errors is asymmetric.
• Judgment criteria must evolve with the business. After a promotional campaign or a routing restructure, the normal behavior profile changes, and the background description in your prompt must change with it.

Anti-bot work isn't a one-off battle—it's an ongoing forensic process. AI turns "logs nobody can read through" into "a list of evidence anyone can read through," and that alone is worth the price of admission.

If you're also looking for a stable LLM gateway to run this kind of log analysis task, check out this platform's registration page: https://api.thistoken.ai/register

---

Tired of juggling provider integrations? Register at https://api.thistoken.ai/register and call every model through one base_url.