Fewer, Closer Examples Beat a Smarter Lookup
Most AI systems answer by first looking things up: past tickets, example cases, documents. Some of what they find is wrong, and some only looks relevant. We tested an idea for making that lookup push the look-alikes away. It did not work. What worked was simpler: read fewer, closer examples. Here is what we tried, what we measured and three lessons we now apply.
01 — the problem
AI answers from what it looks up, and some of that is wrong
Picture a support assistant that sorts a new ticket by finding similar past tickets and copying their category. Two things go wrong. Some past tickets were filed under the wrong category. Others are about a similar product and look right, but are not. We call those look-alikes.
We built a test that recreates both problems. Each prompt shows a few labelled examples for each of ten categories, then asks the model to label a new item. We mislabel 30% of the examples and add categories that closely resemble the right one. The question is how much the model should read before it answers.
02 — the idea
The idea: give look-alikes a negative vote
A standard lookup scores every stored example by how similar it is, and every score counts for the answer. Our idea was to let moderately similar examples count against it: close matches vote yes, moderately similar ones vote no. The eye uses the same trick to sharpen edges, and a recent language-model design does something similar inside attention.
The simpler alternative is to read fewer examples: give full weight to the closest few and exactly zero to the rest. We tested all three approaches on a research model, Memory Mosaics, that makes this lookup step explicit and easy to change.
03 — what happened
Reading fewer examples beat pushing look-alikes away
We trained small models on the noisy task, each method three times, and scored them all on the same held-out prompts. In the hardest setting (30% wrong labels plus look-alike categories), reading fewer examples gained 3.3 points of accuracy over the standard lookup. Pushing look-alikes away gained 1.7. The version that pushed hardest gained almost nothing.
The look-alikes never went away. Whichever method we used, about 30% of each lookup still landed on look-alike examples. The negative votes fell on moderately similar examples of every kind, not on the look-alikes in particular. And on ordinary text prediction, the push-away model learned to use its negative votes and ended about 5% worse than the standard one.
04 — what we apply now
Three practical lessons
These are our reading of the results. They come from small models on a test task, so check them on your own data before relying on them.
1. When some examples are wrong, read fewer of them. One wrong example carries less weight when only a handful of close matches get a vote. For a system that looks up examples or documents before answering, we would try a smaller number of results, or a minimum-similarity cut-off, before adding a cleverer ranking formula.
2. Keeping look-alikes out depends on how items are described. A scoring rule can only decide how far to look. It cannot move a look-alike further away. Whether a similar but wrong item sits close to a new request depends on how items are represented: the embeddings, the metadata, the labels. That is where we would spend the effort.
3. Compare a new idea with the simplest alternative. In our first, cheapest test, the push-away idea beat the standard lookup by 5.5 points. Against the standard alone it looked like a win. Only the comparison with "read fewer", which matched it, showed the gain came from reading less, not from the negative votes.
05 — where this applies
Four places these lessons show up
Any system that looks up examples or documents before it answers faces both problems. These are illustrations of how we would apply the lessons, not results we measured.
What goes wrong: Past tickets were filed under the wrong team, and tickets about a sister product look almost identical.
What to try: Route using the few most similar past tickets above a similarity floor, and add the product to what gets compared.
What goes wrong: Last year's version of a policy reads almost the same as this year's, so the assistant quotes the wrong one.
What to try: Retrieve fewer passages, and tag each with its effective date so outdated versions can be filtered out before ranking.
What goes wrong: Example invoices were categorised by many people over the years, and some are simply wrong.
What to try: Show the model the handful of closest past invoices for each new one, not a long list, so one bad label carries little weight.
What goes wrong: The same item in different sizes or colours looks like a duplicate, and the system merges them.
What to try: Make size, colour and pack count part of how each item is described. Reading more candidates will not separate them.
06 — the conclusion
What an AI reads matters more than how it scores it
Good answers from a lookup come down to two things: reading only the few closest examples, and describing items well enough that the right ones are the closest. A cleverer scoring rule did not add a third.
Wrong examples are a problem of reading too much. When only a few close matches get a vote, one bad example cannot swing the answer.
Look-alikes are a problem of description. If two items look the same to the system, no scoring rule can tell them apart, so the fix belongs upstream, in how items are represented. Our push-away idea tried to fix a description problem by changing the scoring rule, and it could not.
Our tests used small models on a test task, so treat this as a direction to check on your own system, not a rule.