"Which model should we use?" is the most common technical question we get from buyers — usually asked as if it were a religious affiliation. Our answer disappoints ideologues: we don't have a favourite. We have a method. Models change quarterly; the method doesn't.
The uncomfortable truth about leaderboards
Public benchmarks measure how models perform on public benchmarks. Your product will live on your tasks: your documents, your customers' phrasing, your tolerance for a wrong answer. We've seen mid-tier models beat frontier ones on narrow extraction tasks, and we've seen the reverse an hour later on nuanced support conversations. The only leaderboard that matters is one built from your data.
The four axes we actually score
1. Quality on your eval set. Before choosing anything, we build an evaluation suite from real examples of the work — then run the candidates against it and score them. An afternoon of eval-building removes months of opinion-having.
2. Latency. A chat assistant can think for two seconds; a voice agent gets a few hundred milliseconds. Latency requirements alone can eliminate half the field before quality is even discussed.
3. Cost at your volume. The right question isn't "what does a token cost" but "what does a task cost, at ten times today's usage." Frontier quality on routine work is how AI budgets die — most production systems we build route simple requests to small models and reserve the big guns for hard ones.
4. Privacy and deployment. Some data can't leave your walls — healthcare records, financial transactions, anything under strict compliance. That's where open-source models earn their place: run in your VPC or on-premise, the data never travels. The trade is more infrastructure responsibility; sometimes that trade is mandatory.
What this looks like in practice
- Customer-facing chat over your knowledge: a fast commercial model with strong grounding behaviour, behind a retrieval pipeline and evals.
- Agents doing multi-step work with tools: whichever current model tops your tool-use evals — this shifts release to release, which is exactly the point.
- Realtime voice: latency is king; small fast models handle the conversational surface, bigger ones the substance.
- Regulated, on-prem workloads: open-source in your infrastructure, tuned for the task.
Architect for the swap
The real discipline isn't picking today's winner — it's building so that next quarter's winner is an afternoon's change. Model-agnostic plumbing, prompts under version control, and the eval suite as your safety net: swap the engine, rerun the suite, compare numbers, decide on evidence. Products married to one vendor's API in 2024 have already paid for that wedding twice.
Wondering what your task actually needs? That's a 20-minute conversation with an engineer who benchmarks these weekly — ask the studio live or book the call. We'll tell you which model, and more importantly, show you why.
