Here's a story we've now heard from multiple founders, with the names changed: the AI feature ships, users love it, usage grows — and then finance forwards the model-provider invoice with a single question mark. The feature that delighted a hundred users becomes a liability at ten thousand. Nothing was "wrong": every API call did exactly what it should. What was missing was cost engineering.
The only metric that matters: cost per unit of value
Monthly API spend is a useless number on its own. The number that decides your future is cost per conversation (or per document processed, per call handled) — because that's what scales with success. If a support conversation costs you $0.40 in model calls and saves $6 of agent time, growth makes you richer. If a "quick summary" feature costs $0.30 a click and produces no revenue, growth is a countdown.
The five levers, in the order we pull them
1. Right-size the model. The single biggest lever. Most requests inside any real product are simple — classification, extraction, short answers — and small, fast models handle them at a tiny fraction of frontier-model prices. Route by difficulty: cheap by default, expensive by exception.
2. Cache the prompt. Production AI sends the same instructions and knowledge with every single request. Prompt caching means the provider processes that bulk once and reuses it — on a knowledge-heavy system, that's an order-of-magnitude cut on the most expensive part of every call. (Our own homepage AI runs this way.)
3. Put the context on a diet. Every token in costs money and adds latency. Retrieval that fetches five relevant passages beats stuffing forty pages "to be safe." Summarise long histories instead of replaying them. Context discipline is free money.
4. Cache the answers too. Real users ask the same things. If a hundred people ask "what's your refund policy" and the policy hasn't changed, that's one model call and ninety-nine lookups.
5. Meter per feature, alarm on drift. Cost per feature belongs on the same dashboard as latency and errors. A prompt tweak that doubles token usage should page someone — before finance does.
The rule we hold clients to
Know your cost per unit at ten times current volume before the feature ships — not after. It's one spreadsheet afternoon, and it's the difference between an AI budget and an AI surprise. And sometimes the honest output of that spreadsheet is: this step shouldn't use an LLM at all. A regex, a lookup table or a boring workflow tool is free forever — we build those too when they're the right answer.
Want your unit economics sanity-checked? Bring the feature to a free 20-minute call, or ask our homepage AI — cost engineering is one of its favourite subjects, for obvious reasons.
