How I Rebuilt Smart Search for My Side Project: Lessons from Failure and a Better Architecture

Last year I added smart search to my small tool product. The first version took two days to build and ship—and three months later I tore almost all of it down and rewrote it. This article isn't a success story; instead, I'll first lay out the common failure patterns, then offer a path that saves you a lot of pain. If you're an indie developer or on a team of two or three, I hope this helps you avoid the detour I took.

1. Common Failure Patterns

Failure pattern #1: Treating search as "one call to an LLM." The most intuitive solution: user enters a keyword → build a prompt and call GPT → return the answer. In the first week after launch, a user asked "where's the order I bought last week," and the model confidently made up a tracking number. The LLM doesn't have your business data—letting it answer directly is like opening a store that needs no inventory: it can sell anything, and none of it is real.

Failure pattern #2: Stuffing all your data into the context. When the data volume is small, throwing all your products, articles, and orders into the prompt seems to work. Once you exceed tens of thousands of records, token costs explode and requests time out—and after truncation, search quality falls off a cliff. This is trading "brute force" for "intelligence," and the cost curve quickly crushes an indie developer.

Failure pattern #3: Hardcoding model keys into business code. That's exactly what I did in version one. Later, when I hit rate limits and wanted to switch providers, I had to dig through the codebase to find every call site; when I wanted to add retry logic, I had to change each call site individually. After three emergency production fixes, I admitted the architecture was wrong.

Failure pattern #4: No logging, no evaluation. When a user said "the search results are wrong," I couldn't even find out what they had searched for. No logs, no bad-case collection—optimization was pure guesswork.

2. The Core Pain Points

For indie developers building smart search, the core tensions boil down to three:

1. Your data lives in a private database, while the intelligence lives on the model side—the two must be connected securely;
2. Budget and time are limited—you can't build a full-blown search platform like big companies do;
3. Models iterate too fast—today's best model may not be optimal in six months, so switching costs must be low.

The third point is the most easily overlooked, yet it determines your maintenance cost.

3. The Right Architecture

A sensible minimal architecture is "retrieval augmentation (RAG) + a unified AI gateway":

• Data layer: Business data (products, documents, tickets) is synced on a schedule to a vector database;
• Retrieval layer: User queries are vectorized first, then the Top-K relevant chunks are recalled (e.g., 5);
• Generation layer: The recalled chunks serve as context for the LLM to generate an answer with sources;
• Gateway layer: All model calls go through a unified AI API gateway; business code only talks to the gateway.

The key point: there is not a single model name or API key anywhere in the business code. This layer of abstraction is a lifesaver for indie developers.

4. Why a Unified Gateway Significantly Reduces Maintenance Cost

Looking back at failure pattern #3, the root cause was "scattered call logic." The benefits of a unified gateway are very concrete:

1. Configure once, apply everywhere: Switching models or providers only requires changing gateway config—zero changes to business code;
2. Built-in retry and fallback: Rate limits and timeouts automatically switch to a backup model, no hand-written logic at every call site;
3. Centralized key management: Keys live only on the gateway side; code leaks or personnel changes don't compromise security;
4. Call logs for free: Token consumption, response latency, and error rates at a glance; bad cases are traceable;
5. Governable costs: Add caching and quotas at the gateway layer without touching business logic.

For an indie developer, this means one line of integration replaces an entire chunk of infrastructure you'd otherwise maintain yourself.

5. Key Implementation Steps (Process Checklist)

1. Map out searchable data sources and determine sync frequency (real-time / hourly / daily);
2. Choose a vector database (for small scale, pgvector or a cloud service's free tier works fine);
3. Write the data chunking and embedding jobs, and load into the vector database;
4. Register model services in the AI gateway, configuring a primary model + a backup model;
5. Implement the retrieval API: query vectorization → recall Top-5 → assemble the prompt;
6. In the generation layer, require the model to "answer only based on the given content, and say it doesn't know when it can't find the answer," with source citations;
7. Enable call logging and usage alerts on the gateway side;
8. Collect bad cases from the first two weeks and adjust chunk granularity and recall counts accordingly.

The core retrieval logic in pseudocode looks roughly like this:

async def smart_search(query: str):
    # 1. 通过网关调用 embedding，业务代码不含任何模型细节
    vec = await gateway.embed(query)
    # 2. 向量检索
    chunks = vector_db.search(vec, top_k=5)
    if not chunks:
        return "未找到相关内容"
    # 3. 网关自动处理重试、降级、限流
    answer = await gateway.chat(
        system="仅根据以下资料回答，标注来源，资料中没有则明确说不知道",
        context=[c.text for c in chunks],
        user=query,
    )
    return {"answer": answer, "sources": [c.source for c in chunks]}

6. What Changed After Launch

The rewritten version has been running for six months: I've switched model providers twice, each time only changing gateway config and done in ten minutes; during one upstream rate-limiting incident, the backup model took over automatically with zero user impact; and the monthly bill dropped considerably thanks to caching and quota policies at the gateway. More importantly, when a newer, stronger model comes out, I no longer hesitate—because the switching cost is now close to zero.

Smart search itself isn't hard. What's hard is building it as a feature that "won't be obsoleted by model iterations." Build retrieval on a solid foundation, consolidate all model calls into a unified gateway, and you can spend the remaining time on real business growth.

If you're about to get started, register a unified AI API gateway account first and get the infrastructure layer out of the way: https://api.thistoken.ai/register

---

Tired of juggling provider integrations? Register at https://api.thistoken.ai/register and call every model through one base_url.