Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Models

Wentian Wang, Sarthak Jain, Paul Kantor, Jacob Feldman, Lazaros Gallos, Hao Wang

TL;DR

This study modified standardized test questions by replacing a key term with a dummy word along with its definition, and found a substantial reduction in model performance after such replacement, suggesting poor comprehension.

Abstract

We propose MMLU-SR, a novel dataset designed to measure the true comprehension abilities of Large Language Models (LLMs) by challenging their performance in question-answering tasks with modified terms. We reasoned that an agent that "truly" understands a concept can still evaluate it when key terms are replaced by suitably defined alternate terms, and sought to differentiate such comprehension from mere text replacement. In our study, we modified standardized test questions by replacing a key term with a dummy word along with its definition. The key term could be in the context of questions, answers, or both questions and answers. Notwithstanding the high scores achieved by recent popular LLMs on the MMLU leaderboard, we found a substantial reduction in model performance after such replacement, suggesting poor comprehension. This new benchmark provides a rigorous benchmark for testing true model comprehension, and poses a challenge to the broader scientific community.

MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Models

TL;DR

This study modified standardized test questions by replacing a key term with a dummy word along with its definition, and found a substantial reduction in model performance after such replacement, suggesting poor comprehension.

Abstract

We propose MMLU-SR, a novel dataset designed to measure the true comprehension abilities of Large Language Models (LLMs) by challenging their performance in question-answering tasks with modified terms. We reasoned that an agent that "truly" understands a concept can still evaluate it when key terms are replaced by suitably defined alternate terms, and sought to differentiate such comprehension from mere text replacement. In our study, we modified standardized test questions by replacing a key term with a dummy word along with its definition. The key term could be in the context of questions, answers, or both questions and answers. Notwithstanding the high scores achieved by recent popular LLMs on the MMLU leaderboard, we found a substantial reduction in model performance after such replacement, suggesting poor comprehension. This new benchmark provides a rigorous benchmark for testing true model comprehension, and poses a challenge to the broader scientific community.
Paper Structure (18 sections, 5 figures, 14 tables)

This paper contains 18 sections, 5 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. MMLU-SR Dataset
  4. Dataset Construction
  5. "Question Only" Dataset
  6. "Answer Only" Dataset
  7. "Question and Answer" Dataset
  8. Experiments
  9. Evaluation Protocol
  10. Results and Analysis
  11. Conclusion
  12. Acknowledgements
  13. CoT Examples
  14. Examples of ChatGPT-3.5 Response
  15. System Instruction Example
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Illustration of our MMLU-SR testing scenarios. The red-colored and green-colored words represent the original symbols in the MMLU dataset showing in answers and questions, which are replaced in the MMLU-SR dataset with random words followed by their definitions, shown in orange text. The example question from the MMLU dataset is correctly answered by both GPT-3.5-turbo and ChatGPT-4. However, the modified question from the MMLU-SR "Question and Answer" dataset is answered incorrectly by both models.
  • Figure 2: Example ChatGPT-4 output of MMLU-SR 'Question Only".
  • Figure 3: Example ChatGPT-4 output of MMLU-SR "Answer Only".
  • Figure 4: Example ChatGPT-4 output of MMLU-SR "Question and Answer".
  • Figure 5: Comparison of total generated terms (red) and human-modified terms (blue) across 41 subject glossaries