Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?

Kai Yan, Yufei Xu, Zhengyin Du, Xuesong Yao, Zheyu Wang, Xiaowen Guo, Jiecao Chen

TL;DR

This work introduces RoR-Bench, a multimodal benchmark to probe whether cutting-edge LLMs actually reason or merely recite learned solution patterns when faced with elementary problems whose conditions are subtly altered. Across text and vision tasks, the study shows a pervasive recitation tendency, with typical modified-problem performance drops exceeding 50-60% for top models, and only marginal benefit from few-shot learning or forced-correct prompts. The authors analyze contributing factors and find solution-paradigm overfitting as a primary cause, highlighting the limits of current alignment and prompting-based mitigations. The findings have significant implications for evaluating AI reasoning and underscore the need for robust, assumption-resistant reasoning capabilities in future LLMs and multimodal systems.

Abstract

The rapid escalation from elementary school-level to frontier problems of the difficulty for LLM benchmarks in recent years have weaved a miracle for researchers that we are only inches away from surpassing human intelligence. However, is the LLMs' remarkable reasoning ability indeed comes from true intelligence by human standards, or are they simply reciting solutions witnessed during training at an Internet level? To study this problem, we propose RoR-Bench, a novel, multi-modal benchmark for detecting LLM's recitation behavior when asked simple reasoning problems but with conditions subtly shifted, and conduct empirical analysis on our benchmark. Surprisingly, we found existing cutting-edge LLMs unanimously exhibits extremely severe recitation behavior; by changing one phrase in the condition, top models such as OpenAI-o1 and DeepSeek-R1 can suffer 60 percent performance loss on elementary school-level arithmetic and reasoning problems. Such findings are a wake-up call to the LLM community that compels us to re-evaluate the true intelligence level of cutting-edge LLMs.

Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?

TL;DR

This work introduces RoR-Bench, a multimodal benchmark to probe whether cutting-edge LLMs actually reason or merely recite learned solution patterns when faced with elementary problems whose conditions are subtly altered. Across text and vision tasks, the study shows a pervasive recitation tendency, with typical modified-problem performance drops exceeding 50-60% for top models, and only marginal benefit from few-shot learning or forced-correct prompts. The authors analyze contributing factors and find solution-paradigm overfitting as a primary cause, highlighting the limits of current alignment and prompting-based mitigations. The findings have significant implications for evaluating AI reasoning and underscore the need for robust, assumption-resistant reasoning capabilities in future LLMs and multimodal systems.

Abstract

The rapid escalation from elementary school-level to frontier problems of the difficulty for LLM benchmarks in recent years have weaved a miracle for researchers that we are only inches away from surpassing human intelligence. However, is the LLMs' remarkable reasoning ability indeed comes from true intelligence by human standards, or are they simply reciting solutions witnessed during training at an Internet level? To study this problem, we propose RoR-Bench, a novel, multi-modal benchmark for detecting LLM's recitation behavior when asked simple reasoning problems but with conditions subtly shifted, and conduct empirical analysis on our benchmark. Surprisingly, we found existing cutting-edge LLMs unanimously exhibits extremely severe recitation behavior; by changing one phrase in the condition, top models such as OpenAI-o1 and DeepSeek-R1 can suffer 60 percent performance loss on elementary school-level arithmetic and reasoning problems. Such findings are a wake-up call to the LLM community that compels us to re-evaluate the true intelligence level of cutting-edge LLMs.

Paper Structure

This paper contains 27 sections, 5 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. RoR-Bench
  4. Dataset Curation
  5. Dataset Statistics
  6. Evaluations
  7. Text-based Problems
  8. Reliability analysis of the LLM judge
  9. Excluding Auto-Correction of Typos
  10. Vision-based Problems
  11. Is Few-Shot In-Context Learning the Cure?
  12. Overconfidence in Solvability
  13. Why Does Recitation Happen?
  14. Discussion and Conclusion
  15. Challenges of Expanding to Other Languages
  16. ...and 12 more sections

Figures (5)

  • Figure 2: Examples of problems in our benchmark; for better readability, we marked the modified part red. Despite that we build a Chinese benchmark, OpenAI-o1-1217 jaech2024openai, OpenAI-o3 o3 and Gemini 2.5 Pro comanici2025gemini all fail with our English translation for these examples. See Appendix \ref{['sec:icl_prompt']} for another example and Appendix \ref{['sec:example_prompt']} for links to experiment records on the English translation.
  • Figure 3: An illustration of the types of the problem of our dataset, which covers a variety of reasoning problems; we double-checked the problems to ensure the low difficulty of the original ones.
  • Figure : a) Subtly changed condition
  • Figure : a) Subtly changed condition
  • Figure : b) Performance loss due to recitation