"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models

TL;DR

The paper demonstrates that relying on first-token log probabilities to evaluate multiple-choice questions in instruction-tuned LLMs severely misaligns with the models' actual text outputs. By comparing first-token evaluation with a text-output analysis across OpinionQA and MMLU on six models and under varied prompt constraints, it shows mismatch rates often exceed 60%, with refusals and prompt formats driving much of the discrepancy. It introduces a text-output evaluation pipeline using a trained classifier to map model responses to options, revealing that text-based assessments are more robust to prompt perturbations. The findings challenge the validity of first-token MCQ evaluation for realistic interactions and argue for direct text-output inspection in benchmarking and safety-aware deployments.

Abstract

The open-ended nature of language generation makes the evaluation of autoregressive large language models (LLMs) challenging. One common evaluation approach uses multiple-choice questions (MCQ) to limit the response space. The model is then evaluated by ranking the candidate answers by the log probability of the first token prediction. However, first-tokens may not consistently reflect the final response output, due to model's diverse response styles such as starting with "Sure" or refusing to answer. Consequently, MCQ evaluation is not indicative of model behaviour when interacting with users. But by how much? We evaluate how aligned first-token evaluation is with the text output along several dimensions, namely final option choice, refusal rate, choice distribution and robustness under prompt perturbation. Our results show that the two approaches are severely misaligned on all dimensions, reaching mismatch rates over 60%. Models heavily fine-tuned on conversational or safety data are especially impacted. Crucially, models remain misaligned even when we increasingly constrain prompts, i.e., force them to start with an option letter or example template. Our findings i) underscore the importance of inspecting the text output as well and ii) caution against relying solely on first-token evaluation.

Figures (7)

• Figure 1: Example of LLM's mismatch between first-token probability prediction ("C") and text output ("A").
• Figure 2: (a) Mismatch and (b) Refusal rate of different models under the instruction of different constraint levels. The light colour in the mismatch rate indicates the portion of mismatch due to refusal. Results are averaged across 10 runs.
• Figure 3: Result distribution of first token and text output based on example template with (a) "Answer: C" and (b) "Answer: A/B/C".
• Figure 4: Impact of decoding temperature. (a) Consistency. (b) Refusal and Mismatch rate.
• Figure 5: An example survey question with LLM response answer for annotation