Tous les articles

Twelve questions to ask an AI vendor before you sign

We build AI systems for a living, and we still think you should interrogate anyone who does, including us. The questions worth asking an AI vendor check one thing above all: whether the vendor has lived with a system in production. That experience, more than any framework or partnership logo, separates the teams that ship from the teams that demo.

Here are the twelve we would ask in your seat, grouped by what they test, with notes on how a good answer differs from a sales answer.

Production reality

  1. Show me two systems comparable to mine that are in production today. A good answer names them or describes them in verifiable detail: how long they have been live, what volume they handle, what broke in month three. The sales answer is a client logo slide and a polished demo. A vendor with nothing in production is charging you for their education.

  2. Tell me about your last incident. Good vendors describe a concrete failure and what they changed afterwards, without flinching. A vendor who claims to have had no incidents has either no production traffic or no monitoring, and either one should end the meeting.

  3. What happened to the systems you delivered two years ago? You are listening for whether they still run, who operates them now, and whether any client team took one over successfully. Systems that die when the vendor leaves are rentals, whatever the contract says.

Engineering practice

  1. How do you evaluate quality before a release? The good answer involves an eval set built from the client’s real data, scores tracked on every change, and thresholds that block a deploy. The sales answer is "we test extensively". The gap between those two answers is the subject of why evals come before scale.

  2. What happens when the model is wrong? Because it will be. You want to hear about validation rules, fallbacks, review queues, and refusal designed in as a feature. Be wary of anyone whose answer leans on how accurate models have become.

  3. How will we know if the system degrades in month six? Data drifts, document formats change, model providers update. Good vendors describe monitoring, quality signals in production, and a named owner for the dashboard after handover.

Ownership and economics

  1. Who owns the code, and where does it live? The right answer is your organization, in your repositories, from the first week. If the intellectual property terms are vague, or the system only runs on the vendor’s proprietary platform, you are buying a subscription dressed as a build.

  2. What does one request cost, and what does the bill look like at ten times the volume? Teams that run production systems answer with numbers and levers: routing, caching, batching, the mechanics in your AI bill at ten times the volume. The sales answer is that costs are coming down anyway.

  3. What will this cost us to run after you leave? Model bills, hosting, and maintenance hours. A vendor who has never been asked this has never stayed around long enough to find out.

Failure and exit

  1. What would make you tell us to stop the project? A serious vendor has stopping conditions, because some projects should stop. A vendor with no answer has no scoping discipline and every incentive to keep billing.

  2. Of your last ten prototypes, how many reached production? Every vendor’s demos work. The graduation rate is the real record, and the true number is rarely ten. What kills the rest is documented in why most pilots die between POC and production; you want a vendor who can name those causes unprompted.

  3. How do we run this without you? Look for a concrete handover story: documentation, pairing time with your engineers during the build rather than after it, and evidence that your team will have touched the system before the final invoice.

How to read an AI vendor’s answers

Across all twelve, the same rule holds. Good answers are specific, contain numbers, and include failures. Sales answers are fluent and generic, with adjectives where the numbers should be. Fluency costs nothing now, so weigh the specifics.

Two more signals. A vendor who pushes back on your scope in the first meeting is worth more than one who says yes to everything, because the yes-to-everything quote is the one that doubles later. And a vendor who volunteers their weaknesses unprompted has usually been through enough production cycles to know them.

We get asked these questions most weeks, and the market improves every time a buyer asks them. If you want to hear our answers against a project of yours, the custom AI page explains how we work and where a first conversation starts.