The AI gold rush has produced more vendors than builders. Every agency's website now says "AI-first"; very few of their engineers have shipped a language-model product to real users. You can't audit their codebase before signing — but you can ask questions that pretenders can't answer well. Here are the twelve we'd ask if we were buying.
Proof questions
1. "What have you shipped that's running in production right now?" Not case-study PDFs — live systems, with real users, today. A good answer names the product, the scale and what broke along the way. A bad answer is a demo reel.
2. "Do you run any AI products of your own?" Vendors who eat their own cooking learn lessons on their own budget instead of yours. (Ours: a grounded chat engine, a realtime voice stack, and the AI running our own homepage.)
3. "Can I talk to the engineers who'd actually build mine?" If sales won't put engineers on the first call, you've learned how the whole project will go. The people who scope it should be the people who ship it.
Process questions
4. "What happens in week one?" Good answer: access to your data, a thin working slice, the first eval cases. Bad answer: a discovery workshop about a workshop.
5. "How often do I see working software?" The only acceptable cadence is weekly. Slideware is not software.
6. "What happens if I want to stop halfway?" Fair vendors have a real answer: you pay for accepted milestones, you keep everything built so far. If exit terms feel like hostage terms, believe them.
Engineering questions
7. "How will it answer from my knowledge — and what happens when the answer isn't in there?" You're listening for grounding and citations, and for the humility of "it says it doesn't know." Anyone promising an AI that's never wrong has never shipped one.
8. "Show me an evaluation suite from a past project." Evals — scored test questions run before every change — are the difference between engineering and vibes. This single question eliminates half the market.
9. "What will one conversation cost me at ten times my volume?" If they haven't done this arithmetic for previous clients, you'll be the one funding the lesson.
Commercial questions
10. "Who owns the code, the prompts and the data?" The only right answer is: you do, fully, on final payment. No proprietary platforms you're renting forever, no licensing games.
11. "Fixed price or time-and-materials — and why?" The honest answer is "depends on scope clarity" — fixed once scope is nailed, flexible while it isn't. Anyone who answers instantly without asking about your scope is selling a template.
12. "What don't you do?" Real experts have edges. "We do everything" means "we subcontract things and mark them up."
How we score on our own test
We published this list knowing we'd be graded on it — that's rather the point. Our answers live on the FAQ, in the case studies, and in the AI on our homepage, which will answer all twelve questions right now, live. That's question 2 answered, at minimum.
