The 100-example test: proving an AI feature works before you buy it

Every supplier will tell you their AI works. The only way to know is to hand it a set of your own examples, with the right answers written down in advance, and count. This is the cheapest hour you will spend on an AI project, and almost nobody spends it.

The short answer

Collect one hundred real examples from your own business, write the correct answer for each one yourself, and make the supplier run them before you sign anything. Score the result. If the score is worse than the person doing the job today, you have saved yourself a project. If it is better, you now have a number to put in the contract instead of an adjective.

Why a demo proves nothing

A demo is a performance on material the performer chose. It is not dishonest — it is simply the easiest cases, run until they look smooth. Your business is not made of easy cases. It is made of the supplier invoice with the handwritten amendment, the customer who writes half in English and half in Kuwaiti dialect, the form somebody scanned at an angle with a phone.

The gap between demo material and real material is where AI projects quietly fail. You can close it before spending anything, by controlling the material yourself.

What a test set actually is

A test set is two columns. The left column is the input as it really arrives — the actual message, the actual PDF, the actual voice note. The right column is what a correct answer looks like, written by the person who does this work today.

That second column is the part everyone skips, and it is the entire value of the exercise. Writing down the right answer forces you to discover that your team does not always agree on what the right answer is. That disagreement is not a distraction from the AI project. It is the project.

How many examples, and which ones

A hundred is enough to be useful and small enough that you will actually do it. Anthropic’s own guidance on defining success criteria and building evaluations makes the same trade deliberately: more cases graded roughly beats a handful graded perfectly, because the volume is what gives the number meaning.

Choose them like this:

• Sixty ordinary cases, sampled from a normal week rather than hand-picked. Take every fifth item from a real queue.
• Thirty awkward cases — the ones your team complains about. Mixed languages, missing fields, two requests in one message, an attachment that is a photograph of a screen.
• Ten cases where the right answer is “refuse”. A request outside policy, a question needing a human decision, a document that is simply the wrong document.

That last group is the one that separates a serious supplier from a confident one. A system that answers everything is not accurate; it is just never silent.

Scoring without a data-science team

You do not need statistics. You need a rule for what counts as correct, agreed before you look at the output — because a rule invented afterwards will always be the rule that makes the result look acceptable.

Kind of task	What “correct” means	How to score it
Extracting a field	The value matches exactly	Character comparison, no judgement needed
Routing or classifying	Lands in the right category	Count matches against your column
Answering a customer	Contains the right facts, no invented ones	Two people read it, disagreements resolved
Summarising	Nothing important dropped, nothing added	Human read against a checklist

Work down that table before you build. Anything you cannot score, you cannot manage later either.

What an accuracy claim hides

“Ninety-five per cent accurate” is close to meaningless on its own, for three reasons.

The first is the base rate. If ninety per cent of your invoices are from the same six suppliers in the same format, a system can score ninety per cent by handling only those and failing everything else. The headline looks strong and the awkward tenth — the part you wanted help with — is untouched.

The second is which errors those five per cent are. Five per cent of amounts read slightly wrong is a different business from five per cent of documents flagged for a human to check. Same number, opposite consequence. We wrote about drawing that line in reading Arabic invoices with AI.

The third is who measured. An accuracy figure produced on the supplier’s material tells you about their material.

Run it as a sealed test

Give the supplier the left column only. Keep the answers. Let them run the set and return their output, then score it yourself.

This matters more than it sounds. If the supplier can see the answers, the system gets tuned to your hundred examples and will look excellent on them and ordinary on the hundred-and-first. Keeping the answers back is the difference between measuring a system and measuring a rehearsal.

Hold back twenty examples entirely, and run those after go-live. If the score holds, the earlier number was real.

Turning the score into a decision

Compare against the human doing the job now, not against perfection. Google’s Rules of Machine Learning opens on exactly this instinct — its first rule is “Don’t be afraid to launch a product without machine learning”, on the grounds that a simple heuristic often beats an untrained system. Your baseline is that heuristic, and if the AI cannot beat it, that is the finding. People misread invoices too. If you have never measured your current error rate, take fifty of the hundred and check what was actually done with them last month — that is your baseline, and it is usually more forgiving than people expect.

Then decide on the shape of the errors, not the size. Reversible mistakes caught by a later step can be tolerated at a higher rate than a mistake that reaches a customer or a ledger. That is the same test we apply when choosing which process to automate first.

A worked example

A trading company wants an assistant to read supplier invoices and enter three fields into their accounting system: supplier, total, and due date. They pull a hundred invoices from the last two months — sixty ordinary, thirty awkward, ten that should be rejected outright because they are delivery notes, not invoices.

The supplier’s system returns ninety-four of the hundred with all three fields correct. That sounds like a decision made. It is not, until you look at the six.

Four are awkward scans where the total was read one digit short. One is a credit note read as an invoice, which would have posted a payment the wrong way round. One is a delivery note the system confidently processed as an invoice rather than rejecting it.

So the honest reading is different: field extraction is strong, and document-type recognition is not. The build that follows is not “automate invoice entry”. It is “automate invoice entry, with a human confirming any document the system has not seen the shape of before, and any total above a threshold”. That design came out of six examples, and it is the difference between a system that saves time and one that quietly creates reconciliation work.

Put the number in the contract

Once you have a score, it becomes the acceptance criterion: the system is accepted when it reaches an agreed score on a set the supplier has not seen. That single sentence changes the relationship. It replaces an argument about whether it feels better with a measurement you both agreed to in advance.

It also gives you a monitoring plan for free. Re-run the set quarterly. Models change under you, your documents change, your customers change; the set is how you notice. NIST’s AI Risk Management Framework treats measurement as a continuing function rather than a launch gate, and that is the right instinct.

How we run it

In a 30-day pilot the test set is built in week one, before anything is designed, and the honest outcome is sometimes that the score does not justify the build. Both answers are acceptable results for a pilot, and the second one costs you a month instead of a year.

Send us the task you would test first and we will tell you what a hundred examples would need to look like.

Frequently asked questions

What if we do not have a hundred examples?

Then you probably do not have enough volume to justify automating this task yet, which is itself a useful finding. If the volume exists but is not recorded, start recording it for two weeks before you buy anything.

Who should write the correct answers?

The person who does the work today, not the manager and not the supplier. If two people disagree on an answer, write down both readings — those cases are where your process is genuinely ambiguous, and no system will fix that for you.

Can the supplier build the test set for us?

They can help with the format. They should not choose the examples or write the answers. A test set assembled by the party being tested measures cooperation, not accuracy.

How long does this take?

Collecting a hundred real items is typically half a day. Writing the answers is a day spread over a week, because it needs the person who knows, in gaps between their actual job. It is the cheapest week in the project.

Does this apply to chatbots as well as document work?

Yes, with a different right-hand column: the facts that must appear and the things that must never be said. Scoring needs two readers rather than a string comparison, so keep that set smaller — fifty cases is workable.