One Bill We Couldn't Understand — and How We Cut API Costs by 42% with Caching and Routing

一 Wait, no: ## 1. A Bill We Couldn't Understand

At the end of last quarter, our team did an API cost review. The results were a bit awkward: we hadn't changed models, call volume was growing mildly, yet our monthly bill was up 40% quarter-over-quarter. When finance asked why, I couldn't answer.

After pulling the gateway logs and reconciling them line by line, the answer surfaced — our cache hit rate was only 39%. A large number of repetitive requests (similar product Q&As, near-identical document summarization instructions, structured extraction tasks with almost identical formats) were hitting the model at "full price" every time. In other words, we had been paying over and over for the same answers.

We're not alone here. For many teams whose API budgets spiral out of control, the problem isn't that "the unit price is too expensive," but rather not knowing which categories of requests the money is actually going to. Cache hit rate happens to be the most overlooked — and most immediately impactful — lever in the attribution chain.

2. Three Months of Work: Before and After

We split the overhaul into three steps, with the key data changes as follows:

Doubling the hit rate brought more than just savings. Latency dropped nearly 30% because a significant portion of requests never even reached model inference — they hit the cache directly at the gateway layer and returned. Users felt things "got faster" while costs went down — the most satisfying kind of optimization from an efficiency standpoint.

3. Three Concrete Practices

Practice 1: Attribute usage by request fingerprint to see "who is spending repeatedly"

The first step of optimization isn't making changes — it's attribution. At the gateway layer, we computed a "fingerprint" for every request, composed of four elements: model name, system prompt hash, instruction template ID, and user role. Requests with the same fingerprint are potential cache candidates.

The attribution results surprised us: the customer service FAQ scenario accounted for 31% of call volume, but 70% of those questions fell into fewer than 200 fingerprints. In other words, a third of our budget was going to answering the same questions over and over.

Doing this kind of attribution with homegrown scripts is painful — logs are scattered across multiple services with inconsistent formats. After switching to the ThisToken.AI gateway, every request passing through the gateway automatically carries unified metadata tags (project, department, scenario, user role), turning attribution from "half a day of manual reconciliation every week" into "five minutes looking at a dashboard." Unified logging through a managed gateway essentially turns usage attribution from a project management problem into a configuration problem.

Practice 2: Tiered caching strategy — different rules for different scenarios

Once attribution was clear, we configured three tiers of caching by scenario:

Key lesson: don't enable caching globally. Early on, we took the lazy route and enabled semantic caching globally, and users in creative scenarios complained about "repetitive answers." After tiered configuration, hit rates became independently observable per scenario, and troubleshooting got much faster.

Practice 3: Use model whitelists + routing rules to separate "what should be saved" from "what's worth spending on"

Caching solved "repetitive requests," but another category of waste is "using the wrong model." Attribution showed that for low-complexity internal tasks — employees looking up policies or ticket statuses — nearly half the calls were going to the flagship model.

We attacked this on two fronts:

1. Role-based model whitelists: Administrative and operations roles by default can only route to lightweight models; engineering and data teams can unlock the flagship model, but with separate accounting under a distinct budget line.
2. Complexity-based dynamic routing: Each request first goes through lightweight classification (question length, conversation history turns, whether it contains code). Simple requests route to small models, and only tasks classified as complex get upgraded.

We configured these rules through ThisToken.AI's routing governance capabilities — whitelists, budget caps, and fallback strategies are all managed in the console, with no business code changes needed. In the first month after launch, the share of premium model calls dropped from 46% to 21%, while the pass rate on task quality spot checks didn't decline.

4. A Budget Governance Checklist

If you take away just one table, make it this one. Go through it item by item during your monthly review:

A special note on item 6: hit rate is a means, not an end. If semantic cache thresholds are set too loosely, your hit rate looks great but answer quality drops — saving pennies while hurting the experience. Don't skip the monthly spot checks.

5. Final Thoughts

The biggest mindset shift from this review: the main battleground of API cost optimization isn't "switching to a cheaper model" — it's governance at the request layer. Get caching, whitelists, routing rules, and usage attribution right, and a 42% cost reduction is not an aggressive target — and user experience improves rather than degrades.

If you want to start on this, I recommend beginning with unified access through a managed gateway — first fill in the metadata tags for every request and get your attribution dashboard running. Tiered caching and routing strategies will follow naturally. You can register at https://api.thistoken.ai/register and connect your first channel, then let the attribution data accumulate for two weeks. Your first cost review will be far clearer than mine was.

---

Ready to try it yourself? Sign up at https://api.thistoken.ai/register to get your API key and start building.