Offline batch processing, automatic retry on failure, results written to database after validation

tags = gateway.chat(DISH_PROMPT, model="lite", json_mode=True)

**Step 2: Update user profiles offline.** Run a scheduled task every night that feeds the user's orders from the past 30 days to the LLM, outputs a structured taste profile (spice level preference, price range, category tendencies), and writes it to Redis along with the vectors.

**Step 3: Online recommendations are pure computation.** When a user request arrives, perform a vector nearest-neighbor search to get a candidate set, then apply rules: prioritize fast-to-prepare dishes during lunch service, downweight low-stock items, guarantee minimum exposure for new dishes. No LLM calls at all, with P99 latency stable under 100ms.

**Step 4: Connect to a unified AI gateway.** Consolidate all model calls into a single entry point with a tiered model configuration: simple tasks like tag extraction go to a low-cost small model, while profile distillation uses a more capable model, with routing, retries, and usage tracking managed uniformly by the gateway.

## 4. Why a Unified Gateway Saves Massive Maintenance Costs

One of the biggest lessons from the failed version was call logic scattered across three places. After consolidating to a unified AI API gateway:

1. **Change the model in one place, effective everywhere.** Want to switch tag extraction from small model A to cheaper model B? Just modify one routing configuration in the gateway—no business code changes, no redeployment needed.
2. **Keys no longer scattered everywhere.** API Keys are stored only on the gateway side, never exposed in the code repository or the mini program client—a direct step up in security.
3. **Rate limiting, retries, and failover handled centrally.** If a model times out, the gateway automatically switches to the backup model, transparent to the business layer. I don't have to write fault-tolerance logic at every call site.
4. **Transparent billing.** The gateway tracks token consumption by task type, making it obvious which pipeline is burning money, so optimization efforts have a solid basis. After re-evaluation, the monthly cost of offline tasks dropped to less than one-third of the first version.

## 5. Results and a Few Reminders

After the refactored version launched: homepage recommendation responses stabilized at the hundred-millisecond level, recommendation click-through rates improved noticeably, and model call costs became a budgetable fixed expense. Lessons that don't require real data to learn:

- LLMs are extremely good at "understanding and labeling," but "real-time ranking" with them is both expensive and slow;
- All LLM outputs need JSON mode + validation + a fallback;
- For indie developers, time is the most expensive resource—if operational details can be handed off to a gateway, don't write them by hand.

If you're also planning to add AI capabilities to a food ordering or retail mini program, I recommend integrating a unified AI API gateway from day one, separating model management from your business code. You can register for a trial at https://api.thistoken.ai/register and save your energy for the recommendation strategies that truly differentiate you.

---

Tired of juggling provider integrations? Register at https://api.thistoken.ai/register and call every model through one base_url.