Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Changing Answer Order Can Decrease MMLU Accuracy

Vipul Gupta, David Pantoja, Candace Ross, Adina Williams, Megan Ung

TL;DR

This work interrogates the robustness of MMLU-based benchmarking by randomly shuffling answer contents while preserving option labels, revealing consistent accuracy degradation across 10 diverse LLMs. It introduces a formal robustness metric to quantify stability across shuffled versions and demonstrates substantial variance in sensitivity by model type and task category, with problem-solving domains most affected. The findings argue for incorporating randomness and chance-adjusted measures into leaderboard rankings to obtain a truer assessment of capability. The study highlights practical implications for fair evaluation and suggests broader shuffles and category-focused analyses to improve robustness benchmarking.

Abstract

As large language models (LLMs) have grown in prevalence, particular benchmarks have become essential for the evaluation of these models and for understanding model capabilities. Most commonly, we use test accuracy averaged across multiple subtasks in order to rank models on leaderboards, to determine which model is best for our purposes. In this paper, we investigate the robustness of the accuracy measurement on a widely used multiple choice question answering dataset, MMLU. When shuffling the answer label contents, we find that all explored models decrease in accuracy on MMLU, but not every model is equally sensitive. These findings suggest a possible adjustment to the standard practice of leaderboard testing, where we additionally consider the percentage of examples each model answers correctly by random chance.

Changing Answer Order Can Decrease MMLU Accuracy

TL;DR

This work interrogates the robustness of MMLU-based benchmarking by randomly shuffling answer contents while preserving option labels, revealing consistent accuracy degradation across 10 diverse LLMs. It introduces a formal robustness metric to quantify stability across shuffled versions and demonstrates substantial variance in sensitivity by model type and task category, with problem-solving domains most affected. The findings argue for incorporating randomness and chance-adjusted measures into leaderboard rankings to obtain a truer assessment of capability. The study highlights practical implications for fair evaluation and suggests broader shuffles and category-focused analyses to improve robustness benchmarking.

Abstract

As large language models (LLMs) have grown in prevalence, particular benchmarks have become essential for the evaluation of these models and for understanding model capabilities. Most commonly, we use test accuracy averaged across multiple subtasks in order to rank models on leaderboards, to determine which model is best for our purposes. In this paper, we investigate the robustness of the accuracy measurement on a widely used multiple choice question answering dataset, MMLU. When shuffling the answer label contents, we find that all explored models decrease in accuracy on MMLU, but not every model is equally sensitive. These findings suggest a possible adjustment to the standard practice of leaderboard testing, where we additionally consider the percentage of examples each model answers correctly by random chance.
Paper Structure (14 sections, 1 equation, 3 figures, 2 tables)

This paper contains 14 sections, 1 equation, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. MMLU
  4. Metrics
  5. Models
  6. Results
  7. Related Works
  8. LLMs can be Sensitive to Option Order and Label.
  9. Evaluation Dataset Validity.
  10. Conclusion
  11. Appendix
  12. Limitations
  13. Category Wise Analysis
  14. Computation Resources

Figures (3)

  • Figure 1: This figure illustrates the performance of a selection of state-of-the-art models that we tested on the original MMLU (v0) and 2 shuffled versions (v1 and v2). Models are ordered by accuracy drop in 'our metric'. Here '-it' denotes an instruction tuned model. The width of the violin corresponds to the number of subdatasets where the model received a particular score. The white indicator marks the median score for subdataset accuracies.
  • Figure 2: The most and least affected categories of MMLU with our proposed shuffling. The number above each plot signifies percentage change after shuffling. Here '-it' marks instruction finetuned models.
  • Figure 3: Here we show accuracy scores on random categories of MMLU with our proposed shuffling. The number along with each category name signifies the number of questions for that category in MMLU.