One of our software companies ran 18 AI services through 11 API keys. Every month the bill arrived as a single number. It told us how much we spent. It never told us what spent it. After we routed every call through a gateway that tags each request with the service that made it, finding a cost spike became a filter instead of a forensic exercise.
This business has AI woven into everything: researching prospects, validating payment receipts, cleaning up listings, and a dozen jobs besides. Every service constructed its own client with whatever API key sat in its scope. Token usage was logged ad hoc, in whatever shape the author felt like that day. Asking "what did the prospect researcher cost us last week?" meant grepping logs and narrowing to a few suspects. Most of the time we did not ask. An answer was not worth the afternoon.
Then the card started getting charged in the middle of the night.
Nobody was working on the app, so something automated was running. We went to the logs and investigated. Two hours later we found it: a hung process calling the same endpoint over and over. While we searched, it charged us three more times. The damage came to nearly $400 for a loop that produced nothing.
The $400 was a small loss, two hours of an engineer's night cost more. And we spent them only because we could not ask the one question that mattered: which service is this?
We did not build an observability layer. We did not stand up a proxy or shop for an LLM cost analytics vendor. Gateway products already log every request, count tokens, and let you filter by header. Most of that category will tell you in beautiful detail exactly how you spent the money, after you spent it. We wanted the opposite: an answer during the leak, not a postmortem after it.
So we shipped a custom gateway with a module. Every AI service in the codebase now calls it instead of constructing a client directly. It routes the request through our gateway endpoint and attaches the service name as a header, with the metadata we care about alongside it. Nothing else changed. No service rewrote a prompt, a model choice, or its error handling.
Now every dollar carries the name of the thing that spent it. Cost by service, by request, by token. The next spike took two minutes to narrow down. Open the dashboard, filter to a service name, read it off. Same question, same data, different shape.
The incident was the reason we built this. The everyday use turned out to matter more. We can now see what each feature costs to run, so we can tell whether a feature earns its keep and where a smaller model would do the same job. As we have scaled up our LLM usage over the past two years, being able to track down the costs to make sure we are productive has become critical to our budgeting. We are not looking to make our spending go down, but instead make it legible. A bill that we can understand and scrutinize, to make sure we are getting the return on investment we expect.