Three Failure Scenes First

Failure scene one: pinning hopes on the "vendor's cache."

A tool team launched a document summarization service that called the LLM again every time a user opened the same document. The owner heard that some vendors have automatic prompt caching, so they assumed "the platform will save money for me" — and the monthly bill didn't budge. The reason was simple: vendors' automatic caches typically only apply to exactly identical prefixes, but their code prepended a timestamp and random session ID to the head of every prompt — the prefix was never the same, so the cache never hit. If you expect someone else to save you money, first make sure your requests actually "look identical."

Failure scene two: building your own cache at the application layer, but designing the cache key wrong.

Another team did write their own cache layer, but the cache key was the full raw prompt string plus the user ID. Two users asking the same question — cache miss. The same system instruction followed by a different newline character — cache miss. A user asking "how do I use this" versus "how do I use this?" — semantically identical, cache still misses. Worse, they stuffed the entire conversation context into the cache key, and the context included the model's replies from previous turns, so multi-turn conversations almost never hit. They built a cache with a hit rate under 4%, adding a layer of maintenance cost for nothing.

Failure scene three: the cache was built, but nobody could prove it saved money.

The third team got the technology right, with a cache hit rate stably above 60%. But at the quarterly budget meeting, when finance asked "how much did we actually save," they had no answer: the cache lived at the application layer, billing lived on the vendor's side, the logs on both ends didn't match up, and usage attribution was all guesswork. The last mile of budget governance was broken.

All three scenes point to the same conclusion: saving money with caching isn't "flipping a switch" — it's a complete strategy spanning request construction to usage attribution.

The Three Layers of the Correct Path

Layer 1: Make requests "cacheable" — start saving money when you write the code

The upper bound of your cache hit rate is decided the moment you assemble your prompt. Three key points:

1. Stable prefixes: Put system instructions, few-shot examples, and fixed RAG templates at the beginning of the prompt and keep them byte-for-byte identical. Don't embed timestamps, random IDs, request sequence numbers, or anything that changes per request.
2. Sensible cache key design: Don't use the raw string in full as the key. Normalize first (strip whitespace differences, apply a consistent casing policy), then hash "system instructions + core question." For multi-turn conversations, cache only the "stateless part," not the entire history.
3. Distinguish cacheable from non-cacheable requests: Summarization, classification, and fixed-knowledge Q&A are cache gold mines; conversations with personalized context should go down a separate path — don't pollute the cache key space.

Layer 2: Cache and routing governance at the gateway layer — spend money where it counts

The application-layer cache decides "whether to send the request"; the gateway layer decides "which path a request takes." This is also the turning point where many teams go from "out of control" to "in control."

Take the ThisToken.AI gateway as an example: you can configure caching policies and routing rules centrally at the gateway level:

• Model whitelist: Don't rush to let every engineer pick their own model. Configure a whitelist on the gateway that explicitly defines which tiers of models are allowed for which scenarios. Tasks like summarization and tagging are locked to the low-cost tier; only complex reasoning gets the flagship models. On a cache miss, the request won't "casually" land on the most expensive model either.
• Managed channels and fallback routing: When a channel has abnormal latency or quota pressure, the gateway can route requests to a backup managed channel instead of letting requests fail and retry — retry storms are a common invisible killer of blown-up bills.
• Gateway-side cache safety net: Even if some application forgets to implement caching, the gateway layer can provide a second line of defense based on normalized request characteristics, adding a layer of insurance for the whole team's budget.

The value of routing governance: money-saving rules are no longer scattered across each service's code, but centralized, visible, and auditable.

Layer 3: Usage attribution — make the saved money provable

Attribution is what closes the loop on budget governance. If you can't do these three things, even a great cache is just "feeling like you saved":

1. Tag by team/project/feature: Inject attribution tags for every request at the gateway layer, so bills break down by dimension instead of being one big mush.
2. Reconcile cache hits against billing: Record whether each request was a cache hit or a real call, so at month's end you can directly compute the "savings" — and have a common language at the review meeting.
3. Threshold alerts and budget circuit breakers: Set budget caps per attribution dimension. Abnormal growth (e.g., cache key design broken by some code change, a sudden drop in hit rate) triggers automatic alerts or even circuit breaks — instead of a month-end bill surprise.

A Budget Governance Checklist

Conclusion

The phrase "identical prompts shouldn't be billed twice" is half caching technology and half governance: who is allowed to call which model, how the savings get proven, and who notices anomalies. Make requests clean at the application layer, make routing and cache fallback solid at the gateway layer, and make the accounting clear at the attribution layer — only when all three are in place is the budget truly under control.

If your team is preparing to bring AI calls under unified management, you can start by registering a gateway account on ThisToken.AI and getting the whitelist, routing, and usage attribution up and running: https://api.thistoken.ai/register

---

Tired of juggling provider integrations? Register at https://api.thistoken.ai/register and call every model through one base_url.