AI Confidence Scores Depend on the Options You List

01 — the question

Does adding an option change the scores of the others?

Picture a ticket router that scores a message Billing 0.70, Refunds 0.30. Your team sets a rule: above 0.60, route automatically. Next month you add a new category, Gift cards, which has nothing to do with this message and gets almost no score. Billing should still beat Refunds by the same margin. If it does not, your 0.60 rule now means something different, and nobody changed it.

People do not pass this test either. Add a near-copy of one option to a menu and human choices shift. What we wanted to know is whether a model sold as a calibrated decision service behaves better.

02 — how we tested it

2,500 real messages, one option added at a time

We used public collections of real customer-service messages (banking requests and assistant commands) and news headlines. For each message we kept the right answer and the wrong answer the model liked most. Then we added one more option and checked whether the balance between those two moved. The added option was one of five kinds: an unrelated topic, a random category from the same set, one similar to the right answer, one similar to the wrong answer, or a reworded copy of the wrong answer.

We also grew the list from 2 options to 255, filling it either with unrelated labels or with other labels from the same domain. Because the model does not give identical answers to identical calls, we ran every list twice and only counted movement beyond that normal wobble. We wrote the tests and the pass marks down before the first call. The whole run cost $6.79.

03 — what we found

Every added option moves the balance, and longer lists move it more

Even an option nobody picks moves the scores. An unrelated news topic, which received at most 0.1% of the probability, still shifted the balance between the right and wrong answer by more than the model's own run-to-run wobble. Similar options moved it more, up to twice the wobble, and a reworded copy of the wrong answer moved it most, up to three times.

The direction makes sense. An option similar to the wrong answer draws score away from it, which slightly helps the right answer. An option similar to the right answer did not measurably hurt it. A reworded copy of the wrong answer is the clearest case, shown below.

The longer the list, the further scores drift, with no plateau. Each doubling of the list added a similar amount of drift. Unrelated filler drifted most and, past about 16 options, started pulling scores away from the right answer; at 255 options it took 7% of the probability and cut accuracy from 85% to 81%. Filler from the same domain drifted less and cost about 1.5 points of accuracy at 255.

Questions in the same call do not interfere. The service lets you ask several questions about one message in a single call. Adding up to 128 other questions, including leading ones, changed the score by less than 0.001. The coupling stays inside each question's own list.

04 — the fix and its price

Stable scores are available, but they pick worse

Instead of one "pick one of these" question, you can ask a separate yes/no question for each option: "Is this a billing request?", "Is this a refund request?". Those scores barely moved when we added options, up to 128. But taking the option with the highest yes-score as the answer was 8.7 points less accurate, and the gap widened with long lists (69% against 85% at 128 options).

We saw the same trade-off in two models we ran ourselves. One that scores each option on its own had almost perfectly stable scores but lost accuracy fastest as options were added. One that reads the whole list as part of its prompt drifted roughly two to three times more than the hosted model. The hosted model sat in between, and kept the best accuracy on long lists. Looking at every option together is what makes a model pick well, and also what makes its scores depend on the list.

05 — what we apply now

Five practical lessons

These come from one model version on short text classification with plain label names. Other models and tasks may behave differently, so check on your own system.

1. Compare scores only within one list. A 0.70 from a 10-option list and a 0.70 from a 40-option list are not the same thing. When you add, remove or rename categories, re-tune your thresholds on the new list.

2. Remove near-duplicate options. Two names for the same thing split the score between them and change the balance with everything else. One flipped the top answer in about 1 in 7 messages.

3. Do not pad the list with options that cannot apply. Unrelated options looked harmless but pulled scores away from the right answer and cost accuracy on long lists.

4. Pick with one question, compare with yes/no. Use a single "pick one" question when you need the best answer. Use one yes/no question per option when you need scores that mean the same thing across different lists, and accept lower accuracy.

5. Bundle independent questions, and average when the number matters. Packing several questions into one call is safe. But identical calls differ slightly, and about 1% of top answers flip between repeats, so average a few calls when the exact score feeds a decision.

06 — the conclusion

A score is relative to the list it came from

For picking an answer, this rarely matters: one added option changed the top answer in about 1 in 18 messages, and accuracy held up to a few dozen options from the same domain. It matters when scores are treated as stable numbers, such as thresholds tuned once and reused, or dashboards that compare confidence over time while the category list changes underneath them.

Keep the option list fixed, keep it clean, and choose the question type by what you need: the best answer, or a score you can compare.

The full paper, with every test, interval and table, is on arXiv.