How to choose an AI vendor: questions that matter

Short answer: choose an AI vendor in three moves. First, weight your requirements before you see any demos. Second, keep must-haves to a handful, and treat a failed must-have as disqualifying, however strong the rest of the score is. Third, check whether a small change in your weights would flip the winner. If it would, the decision isn’t made yet, and you know which question settles it.

Why do vendor evaluations go wrong?

Most go wrong before scoring starts. Teams watch demos first and write requirements afterwards, so the requirements quietly describe the vendor they already liked. Others list thirty “must-haves”, which means nothing is really a must-have and the loudest feature wins.

The fix is ordering. Decide what matters, and how much, while every vendor is still a name on a list.

Must-have: a requirement a vendor has to meet to stay in the running, whatever else it scores. If data must stay in your region and a vendor can’t do that, a perfect demo doesn’t change the answer.

What should an AI agent platform be scored on?

Phrase each requirement as a question a vendor has to answer with evidence, not a feature they can tick. For an AI agent platform, start with these:

Data residency. Can our data stay in our region?
Access. Is there single sign-on, with roles we control?
Audit trail. Is every agent action recorded in a log we can export?
Integration. Does it connect to our ERP natively, or through something we’d have to build and maintain?
Human approval. Can a person approve actions above a threshold we set?
Model choice. Can we choose or change the underlying model?
Cost control. Are there spend caps and usage reporting by workload?
Quality. How is agent output evaluated and monitored in production?
Assurance. Can they show independent security assurance?

Weight each one from 1 to 5. Flag no more than three as must-haves. In our experience, three is enough to protect you without turning the shortlist into a single name before scoring starts.

A worked example: the highest score that loses

Fictional vendors and scores, chosen to show the method. A finance team scores three AI agent platforms on nine weighted requirements, with audit trail, data residency and single sign-on as must-haves.

Vendor	Weighted score	Must-haves	Result
Vendor C	79.5%	Fails audit trail	Out
Vendor B	78.5%	All met	Recommended
Vendor A	77.0%	All met	Close second

Vendor C has the highest total and still loses, because it can’t export a complete record of what its agents did. For a finance team that answers to auditors, that settles it.

Now the sensitivity check. Vendor B leads Vendor A by 1.5 points. Drop the weight on native ERP integration from 4 to 3 and Vendor A wins. That doesn’t mean the scorecard failed. It means the decision depends on one question, so ask it directly: how much will integration through a connector really cost us to build and run?

A scorecard’s most useful output is not the winner. It is the one question that would change the winner.

What should you ask each vendor before you sign?

Look at your scorecard for three kinds of gaps, and turn each into a question:

Unscored items. Anything you couldn’t score is something you don’t know yet.
Low scores on heavily weighted items. Ask what is on the roadmap, and by when, in writing.
Scores with no evidence. A claim from a demo is not evidence. Ask for documentation, a reference customer, or a proof of concept on your own data.

Put must-have questions first. Run reference calls with customers in your industry, and ask them what surprised them after go-live, not what they liked about the demo.

Key takeaways

Weight requirements before demos, so the scorecard describes your needs, not a vendor’s strengths.
Keep must-haves to about three, and let a failed must-have disqualify.
Phrase requirements as questions that need evidence.
Run a sensitivity check. If one weight flips the winner, that is your deciding question.
Turn unscored, low and unevidenced items into the questions you ask before signing.

Score your own shortlist

The Vendor Selection Scorecard starts with requirement sets for AI agent platforms and ERP. It scores up to four vendors, flags must-have failures, tells you whether one weight change would flip the winner, and writes the questions to ask each vendor. It’s free and runs in your browser.

Frequently asked questions

How do you choose an AI agent vendor?

Weight your requirements before demos, mark a few as must-haves, score each vendor with evidence, disqualify any vendor that fails a must-have, and check whether a small change in weights would change the winner.

How many must-have requirements should a vendor evaluation have?

As a rule of thumb, no more than three. More than that usually means preferences have been labelled as requirements, and the shortlist shrinks before you’ve compared anything.

What is a sensitivity check in vendor selection?

It tests whether moving any single requirement’s weight by one point would change which vendor wins. If it would, that requirement is the question your decision really depends on.

What should AI vendors be asked about governance?

Ask whether every agent action is in an exportable audit log, whether a person can approve actions above a threshold you set, how output quality is monitored, and whether your data can stay in your region.