Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics

Hamed Mahdavi, Alireza Hashemi, Majid Daliri, Pegah Mohammadipour, Alireza Farhadi, Samira Malek, Yekta Yazdanifard, Amir Khasahmadi, Vasant Honavar

TL;DR

This paper critically evaluates whether state-of-the-art LLMs genuinely perform Olympiad-level mathematical reasoning. By combining expert human evaluation on 455 IMO Shortlist problems with automatic reasoning-soundness analyses and LLM-based judging experiments, it reveals a substantial gap between correct final answers and sound, verifiable reasoning. The study introduces a fallacy-based taxonomy for common errors and demonstrates that LLMs frequently rely on heuristics rather than formal proof, and that their ability to verify reasoning is unreliable. The findings call for benchmarks and training paradigms that prioritize logical rigor and verifiability of reasoning, not just final answer accuracy, to advance mathematical problem-solving with LLMs.

Abstract

Recent advances in large language models (LLMs) have shown impressive progress in mathematical reasoning tasks. However, current evaluation benchmarks predominantly focus on the accuracy of final answers, often overlooking the crucial logical rigor for mathematical problem solving. The claim that state-of-the-art LLMs can solve Math Olympiad-level problems requires closer examination. To explore this, we conducted both qualitative and quantitative human evaluations of proofs generated by LLMs, and developed a schema for automatically assessing their reasoning capabilities. Our study reveals that current LLMs fall significantly short of solving challenging Olympiad-level problems and frequently fail to distinguish correct mathematical reasoning from clearly flawed solutions. Our analyses demonstrate that the occasional correct final answers provided by LLMs often result from pattern recognition or heuristic shortcuts rather than genuine mathematical reasoning. These findings underscore the substantial gap between LLM performance and human expertise in advanced mathematical reasoning and highlight the importance of developing benchmarks that prioritize the soundness of the reasoning used to arrive at an answer rather than the mere correctness of the final answers.

Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematics

TL;DR

This paper critically evaluates whether state-of-the-art LLMs genuinely perform Olympiad-level mathematical reasoning. By combining expert human evaluation on 455 IMO Shortlist problems with automatic reasoning-soundness analyses and LLM-based judging experiments, it reveals a substantial gap between correct final answers and sound, verifiable reasoning. The study introduces a fallacy-based taxonomy for common errors and demonstrates that LLMs frequently rely on heuristics rather than formal proof, and that their ability to verify reasoning is unreliable. The findings call for benchmarks and training paradigms that prioritize logical rigor and verifiability of reasoning, not just final answer accuracy, to advance mathematical problem-solving with LLMs.

Abstract

Recent advances in large language models (LLMs) have shown impressive progress in mathematical reasoning tasks. However, current evaluation benchmarks predominantly focus on the accuracy of final answers, often overlooking the crucial logical rigor for mathematical problem solving. The claim that state-of-the-art LLMs can solve Math Olympiad-level problems requires closer examination. To explore this, we conducted both qualitative and quantitative human evaluations of proofs generated by LLMs, and developed a schema for automatically assessing their reasoning capabilities. Our study reveals that current LLMs fall significantly short of solving challenging Olympiad-level problems and frequently fail to distinguish correct mathematical reasoning from clearly flawed solutions. Our analyses demonstrate that the occasional correct final answers provided by LLMs often result from pattern recognition or heuristic shortcuts rather than genuine mathematical reasoning. These findings underscore the substantial gap between LLM performance and human expertise in advanced mathematical reasoning and highlight the importance of developing benchmarks that prioritize the soundness of the reasoning used to arrive at an answer rather than the mere correctness of the final answers.

Paper Structure

This paper contains 14 sections, 47 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. General Workflow
  4. Problem Selection Rationale
  5. Classifying the Failure Modes of LLM Solutions
  6. Data Annotation Processs
  7. Human Evaluation Results
  8. Automatic Evaluation Results
  9. Conclusion
  10. Appendix
  11. Illustrative Examples for Common LLM Fallacies
  12. Real Examples for Common LLM Fallacies
  13. Evaluation Prompts
  14. Additional Notes

Figures (4)

  • Figure 1: LLMs often fail to generate logically sound solutions for challenging problems.
  • Figure 2: Relative frequencies of each fallacy among the LLM-generated solutions for each model.
  • Figure 3: Relative frequencies of each fallacy in LLM-generated solutions, comparing questions with and without a final answer.
  • Figure 4: Relative frequencies of each fallacy in LLM-generated solutions among different topics