Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AMO-Bench: Large Language Models Still Struggle in High School Math Competitions

Shengnan An, Xunliang Cai, Xuezhi Cao, Xiaoyu Li, Yehao Lin, Junlin Liu, Xinxuan Lv, Dan Ma, Xuanlin Wang, Ziwen Wang, Shuang Zhou

TL;DR

AMO-Bench tackles benchmark saturation by introducing 50 original, IMO-level problems designed for final-answer grading and automated evaluation. The authors establish a rigorous four-stage construction pipeline (data creation, quality, originality, and difficulty reviews) and provide detailed statistics on categories and solution-lengths, plus a mixed grading approach with high reported reliability. Experimental results across 26 LLMs reveal a top accuracy of 52.4% and substantial token usage, underscoring persistent gaps in advanced mathematical reasoning. The work also highlights promising scaling trends with test-time compute, suggesting clear paths for improving reasoning capabilities in future models and providing a valuable resource for ongoing research.

Abstract

We present AMO-Bench, an Advanced Mathematical reasoning benchmark with Olympiad level or even higher difficulty, comprising 50 human-crafted problems. Existing benchmarks have widely leveraged high school math competitions for evaluating mathematical reasoning capabilities of large language models (LLMs). However, many existing math competitions are becoming less effective for assessing top-tier LLMs due to performance saturation (e.g., AIME24/25). To address this, AMO-Bench introduces more rigorous challenges by ensuring all 50 problems are (1) cross-validated by experts to meet at least the International Mathematical Olympiad (IMO) difficulty standards, and (2) entirely original problems to prevent potential performance leakages from data memorization. Moreover, each problem in AMO-Bench requires only a final answer rather than a proof, enabling automatic and robust grading for evaluation. Experimental results across 26 LLMs on AMO-Bench show that even the best-performing model achieves only 52.4% accuracy on AMO-Bench, with most LLMs scoring below 40%. Beyond these poor performances, our further analysis reveals a promising scaling trend with increasing test-time compute on AMO-Bench. These results highlight the significant room for improving the mathematical reasoning in current LLMs. We release AMO-Bench to facilitate further research into advancing the reasoning abilities of language models. https://amo-bench.github.io/

AMO-Bench: Large Language Models Still Struggle in High School Math Competitions

TL;DR

AMO-Bench tackles benchmark saturation by introducing 50 original, IMO-level problems designed for final-answer grading and automated evaluation. The authors establish a rigorous four-stage construction pipeline (data creation, quality, originality, and difficulty reviews) and provide detailed statistics on categories and solution-lengths, plus a mixed grading approach with high reported reliability. Experimental results across 26 LLMs reveal a top accuracy of 52.4% and substantial token usage, underscoring persistent gaps in advanced mathematical reasoning. The work also highlights promising scaling trends with test-time compute, suggesting clear paths for improving reasoning capabilities in future models and providing a valuable resource for ongoing research.

Abstract

We present AMO-Bench, an Advanced Mathematical reasoning benchmark with Olympiad level or even higher difficulty, comprising 50 human-crafted problems. Existing benchmarks have widely leveraged high school math competitions for evaluating mathematical reasoning capabilities of large language models (LLMs). However, many existing math competitions are becoming less effective for assessing top-tier LLMs due to performance saturation (e.g., AIME24/25). To address this, AMO-Bench introduces more rigorous challenges by ensuring all 50 problems are (1) cross-validated by experts to meet at least the International Mathematical Olympiad (IMO) difficulty standards, and (2) entirely original problems to prevent potential performance leakages from data memorization. Moreover, each problem in AMO-Bench requires only a final answer rather than a proof, enabling automatic and robust grading for evaluation. Experimental results across 26 LLMs on AMO-Bench show that even the best-performing model achieves only 52.4% accuracy on AMO-Bench, with most LLMs scoring below 40%. Beyond these poor performances, our further analysis reveals a promising scaling trend with increasing test-time compute on AMO-Bench. These results highlight the significant room for improving the mathematical reasoning in current LLMs. We release AMO-Bench to facilitate further research into advancing the reasoning abilities of language models. https://amo-bench.github.io/

Paper Structure

This paper contains 29 sections, 3 equations, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. AMO-Bench
  3. Construction Pipeline
  4. Data creation.
  5. Quality review.
  6. Originality review.
  7. Difficulty review.
  8. Dataset Statistics
  9. Problem categories.
  10. Length distribution of human-annotated solutions.
  11. Grading Method
  12. Grading accuracy.
  13. Experiments
  14. Experimental Setup
  15. Models.
  16. ...and 14 more sections

Figures (9)

  • Figure 1: Performance of top-tier reasoning models on AMO-Bench as well as existing competition-level math benchmarks. Except for the results on AMO-Bench, all other results are sourced from longcatflashthinking.
  • Figure 2: The construction and grading pipeline of AMO-Bench.
  • Figure 3: Basic statistics of AMO-Bench. (a) The distribution of problem categories in AMO-Bench. (b) The distribution of human-annotated solutions in AMO-Bench as well as the comparison with MATH500 and AIME24.
  • Figure 4: The AVG@32 performance of various LLMs on AMO-Bench.
  • Figure 5: The AVG@32 performance of LLMs vs. the average model output length.
  • ...and 4 more figures