One Late-Night Incident Investigation Triggered by a Single SMS

One Thursday morning last month, I was woken up by a cost alert SMS from our cloud provider: that day's API consumption had reached 6x the daily average and was still climbing. As the lead of a small team maintaining three AI feature modules simultaneously, the first question I faced wasn't "how much money" — it was "which one."

The essence of a sudden spike in token usage is that the correspondence between usage and business actions is broken. If you don't know which feature, which model, or which type of request is burning through your budget, all you can do is watch the curve climb. This post is a retrospective of the complete investigation path we went through that night, and how we later compressed troubleshooting time from hours down to minutes.

Step 1: Characterize First — Is It Real Business Growth, or Is Something Broken?

A spike doesn't equal an anomaly. We built a simple triage process:

That night we fell into the third category: tokens were up 5x, but successfully completed business requests had only grown 12%. This "scissors gap" was the key piece of evidence.

Step 2: Attribute — Break One Ledger into Many

Looking back, our biggest handicap was this: all calls shared a single API key under one vendor account. The usage dashboard showed only one merged curve, so attribution amounted to blind guessing.

The investigation path should peel back three layers:

1. Split by caller: Which application, which module contributed it?
2. Split by model: Did requests that should have gone to a small model drift to the flagship model?
3. Split by request characteristics: Is it input tokens or output tokens that are ballooning? Is there abnormally repetitive content in the prompts?

The root cause we finally identified was mundane: a teammate had launched a new feature without setting an output length cap. It hit a long-document summarization scenario, and a single response generated over 8,000 output tokens. On top of that, the failure retry logic was written as unbounded exponential backoff. The two issues stacked together, and the bill exploded. Locating the cause took 3 hours 40 minutes — 3 of which were spent "guessing which module," while actually reading the code took only 40 minutes.

Step 3: Govern — Three Ways to Make Attribution Fast

After the incident, we did three things, each with quantifiable returns.

Method 1: Split Keys and Quotas by Team/Feature

Allocate an independent API key to each feature module, each with its own daily quota and monthly budget. This isn't bureaucracy: after the split, when an anomaly occurred again, locating it went from "3 hours of guessing" to "one glance at which key crossed the line, locked down within 10 minutes." For a three-person team, the troubleshooting time saved in a single incident roughly equals one developer-day of output.

Method 2: Model Whitelist + Routing Governance

We consolidated available models into the gateway's whitelist. Code no longer hardcodes model names; instead, it declares a task tier (lightweight/standard/complex), and the router decides which model handles it. There are two benefits:

• It prevents "model drift" — no one can quietly point a summarization task at the flagship model in code;
• When switching vendors or adjusting pricing, you just change the routing configuration — no need to modify code one place at a time.

After the switch, we sampled 30 days of requests: about 61% of calls were lightweight tasks, most of which had previously misused the flagship model. This routing consolidation alone cut total token costs by roughly 40%, with no business team perceiving any quality change — because lightweight tasks never needed a large model in the first place.

Method 3: Managed Channels + Usage-Dimension Observability

When you build your own multi-vendor integrations, usage data is scattered across each vendor's backend with inconsistent measurement units (some by characters, some by tokens). After switching to a managed gateway (we use the ThisToken.AI gateway), all calls pass through a unified entry point, naturally giving us unified usage dimensions:

• Token consumption per key, per model, per time window — all visible at a glance;
• Budget alerts can be set per dimension, instead of one global red line;
• When an anomalous spike occurs, you look at the grouped curves before diving into code — the investigation path converges instead of diverging.

When the same "unbounded output + infinite retries" issue recurred after governance, we locked down the module within 15 minutes of the alert firing — because the alert itself carried the key attribution information.

Appendix: Our Budget Governance Checklist

After rolling out governance, we distilled the lessons into a checklist you can apply directly:

Running the Efficiency Numbers

Comparing the state during that incident with the post-governance state:

• Troubleshooting time: 3 hours 40 minutes → located within 15 minutes, roughly 95% saved;
• Token cost: total cost down ~40% after tiered routing;
• Configuration changes: switching to or adding a vendor went from code changes + regression testing (about half a day) to changing one routing config (minutes);
• Alert false positives: with a single global red line, alerts were either too late or too noisy; with multi-dimensional alerts, every alert maps directly to a responsible module.

For a three-person team, these numbers translate into concrete returns: roughly two to three weeks of troubleshooting and configuration work saved per year, plus a sizable discount on the token bill — and the team can finally sleep through the night.

Conclusion

A token spike isn't scary. What's scary is a bill that's one structureless aggregate curve. Key splitting, model whitelists, tiered routing, and a managed gateway with unified usage are all, at their core, groundwork for "fast attribution when an anomaly happens." The cost of governance is a one-time expense; the savings in troubleshooting compound with every incident.

If your team is still sharing one key and staring at one merged curve, I suggest rebuilding your usage ledger starting with a unified gateway — ThisToken.AI offers integrated routing governance with a gateway, model whitelist, and managed channels. Registration here: https://api.thistoken.ai/register. When the next 3 a.m. alert SMS arrives, you'll be glad you drew that decision tree in advance.

---

Ready to try it yourself? Sign up at https://api.thistoken.ai/register to get your API key and start building.