A Bill That Raised Questions
Last month, while doing usage attribution, I noticed something strange: one of our team's internal document Q&A services handles about 4,000 calls per day, and roughly 35% of those requests have nearly identical prompt prefixes—the same "system prompt + retrieval context template," with only the user's specific question appended at the end.
In other words, every working day we were repeatedly paying for the same prefix of over ten thousand tokens, and repeatedly waiting for it to be processed. Based on our service duration estimates at the time, this "repetition" overhead translated to nearly 40 minutes of accumulated response latency per backend engineer per day—hundreds of person-hours of hidden cost over a month.
This is not an isolated case. Hidden inside many teams' bills is an invisible "repeater."
Why Caching Saves Money and Time: Three Layers
1. Provider-Side Prompt Caching
Major LLM API providers have gradually rolled out prompt caching mechanisms: when the same prefix hits the cache, input tokens are billed at a discount, and time-to-first-token (TTFT) drops significantly. But the rules vary—some require the prefix to reach a minimum token count, some have cache TTL windows, and some require explicit cache breakpoints. This raises a governance question: should your application-layer code contain adaptation logic for every provider's caching rules?
Our answer is no. We route all requests through the ThisToken.AI gateway, which handles each provider's caching capabilities at the gateway layer. Application code only sends standard requests, and cache hits are transparently visible on the gateway side. Before the change, switching model providers meant re-reading the caching docs and modifying the calling code; after the change, switching models has zero impact on the application's caching behavior.
2. Gateway-Level Response Caching: Identical Requests Are No Longer Billed at All
Provider prompt caching only reduces input cost; it doesn't eliminate duplication. For highly idempotent scenarios (document summarization, classification and tagging, fixed-knowledge Q&A), a more thorough approach is response caching at the gateway layer: completely identical requests return the cached answer directly, the upstream call never happens, and thus it isn't billed.
After configuring response caching on ThisToken.AI for our document summarization endpoint, the before-and-after comparison was striking:
- Upstream call volume dropped by about 28%, with spending dropping proportionally;
- The endpoint's P95 response time went from several seconds down to milliseconds;
- Rate-limiting alerts during peak hours dropped to zero.
Of course, response caching is only suitable for scenarios where "deterministic output is acceptable." Don't enable it for conversational or creative requests, or users will get stale answers. That's also why we made it a per-endpoint, per-channel configuration item rather than a global switch.
3. Routing Governance: Directing Cache-Friendly Requests to the Right Model
The third layer is routing. Caching benefits are coupled with model selection: if the same prompt prefix goes to model A today and model B tomorrow, neither cache will hit. We did two things in ThisToken.AI's routing governance:
Model whitelist: The document Q&A service is only allowed to route to two whitelisted models, preventing an experimental switch from invalidating the entire cache. The "hit rate dropped by 30%" lesson mentioned earlier was caused by someone temporarily switching models and not switching back.
Managed channel grouping: Production traffic and experimental traffic are split into different channels, so cache pollution and cost fluctuations from experimental requests don't propagate to the production bill, and usage attribution stays independent for each.
Three Methods for Controlling Budget and Usage Attribution
In practice, we settled on three methods for your reference:
Method 1: Set budget caps and circuit breakers per channel. Create managed channels per project on ThisToken.AI, set a monthly budget cap for each, and automatically trip a circuit breaker with alerts when the cap is exceeded. This is much healthier than discovering overspending on the month-end bill—during one prompt regression bug, the channel circuit breaker stopped a retry storm within two hours that could have burned through the entire monthly budget.
Method 2: Tag requests so bills are attributable. Every request carries team/project/feature tags, and the gateway generates usage reports by dimension. Once attribution was in place, "which business is burning money and what the cache hit rate is" became immediately clear, and budget discussions went from guessing to looking at data.
Method 3: Tiered routing + cache-first strategy. Divide requests into three tiers: cacheable, low-latency-sensitive requests go to lightweight models with response caching; requests requiring strong reasoning go to flagship models; experimental traffic goes to separate channels. Routing rules are centralized in gateway configuration rather than scattered across code.
Budget Governance Checklist
| Checklist Item | Ungoverned State | Post-Governance Target |
|---|---|---|
| Are identical prompts billed repeatedly | No visibility, all forwarded upstream | Response caching covers idempotent scenarios, duplicate requests no longer billed |
| Do prompt prefixes hit provider caches | Depends on each provider's rules, breaks on switching | Gateway handles integration uniformly, model switching doesn't break caching strategy |
| Can models be switched freely | Anyone can temporarily switch models, invalidating all caches | Whitelist + change tracking |
| How is budget overrun discovered | Month-end bill review | Channel-level caps + circuit breakers + real-time alerts |
| Usage attribution | One aggregate bill, no breakdown | Multi-dimensional tagged reports, cache hit rate queryable |
| Experimental traffic isolation | Mixed with production, mutual pollution | Independent managed channels, cost and caching don't affect each other |
A Word of Caution
A caching strategy isn't "enable it and profit." Two things must be done upfront: first, clearly identify which endpoints produce idempotent, cacheable outputs; second, establish hit-rate monitoring—an abnormal drop in hit rate is often a signal that prompts were unintentionally changed or models were switched, which is both a cost issue and a stability signal.
After a month of governance, our overall document Q&A spending dropped by about 30%, and endpoint latency dropped by an order of magnitude. For budget-constrained teams, "not billing repeatedly for identical prompts" may offer the best ROI of any optimization: it requires no model changes and no business logic changes—just configuring the rules correctly at the gateway layer.
If you also want to start with caching and routing governance at the gateway layer, you can register a channel on ThisToken.AI and give it a try: https://api.thistoken.ai/register
---
Ready to try it yourself? Sign up at https://api.thistoken.ai/register to get your API key and start building.
Bạn muốn thử Token.AI?
Tạo API Key cấp dự án, bật kênh trong bảng điều khiển và định cấu hình định tuyến, ngân sách và nhật ký kiểm tra.
注册 ThisToken.AI 并获取 API Key