UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models

TL;DR

UGMathBench tackles the need for a fair, comprehensive, and dynamic evaluation of undergraduate-level mathematical reasoning by introducing a 5,062-problem benchmark spanning 16 subjects and 111 topics, with 3 randomized versions per problem. It defines effective accuracy $\text{EAcc}$ and reasoning gap $\Delta$ to quantify true reasoning and robustness across problem variations, and assesses 23 LLMs, finding a peak $\text{EAcc}$ of $56.3\%$ (OpenAI-o1-mini) with substantial $\Delta$ across models. The results reveal a notable gap between top proprietary and open-source models, substantial variability across subjects and difficulty levels, and a predominance of calculation errors in failures, highlighting the need for truly scalable 'large reasoning models' with $\Delta=0$. By providing open data and evaluation code, UGMathBench aims to drive advances in undergraduate mathematical reasoning and robust problem solving in LLMs.

Abstract

Large Language Models (LLMs) have made significant strides in mathematical reasoning, underscoring the need for a comprehensive and fair evaluation of their capabilities. However, existing benchmarks often fall short, either lacking extensive coverage of undergraduate-level mathematical problems or probably suffering from test-set contamination. To address these issues, we introduce UGMathBench, a diverse and dynamic benchmark specifically designed for evaluating undergraduate-level mathematical reasoning with LLMs. UGMathBench comprises 5,062 problems across 16 subjects and 111 topics, featuring 10 distinct answer types. Each problem includes three randomized versions, with additional versions planned for release as leading open-source LLMs become saturated in UGMathBench. Furthermore, we propose two key metrics: effective accuracy (EAcc), which measures the percentage of correctly solved problems across all three versions, and reasoning gap ($Δ$), which assesses reasoning robustness by calculating the difference between the average accuracy across all versions and EAcc. Our extensive evaluation of 23 leading LLMs reveals that the highest EAcc achieved is 56.3\% by OpenAI-o1-mini, with large $Δ$ values observed across different models. This highlights the need for future research aimed at developing "large reasoning models" with high EAcc and $Δ= 0$. We anticipate that the release of UGMathBench, along with its detailed evaluation codes, will serve as a valuable resource to advance the development of LLMs in solving mathematical problems. Codes and data are available at https://github.com/YangLabHKUST/UGMathBench

Figures (13)

• Figure 1: Overview of UGMathBench. UGMathBench is a diverse and dynamic benchmark specifically designed for evaluating undergraduate-level mathematics with LLMs, covering 16 distinct subjects and featuring 10 different answer types. Each problem contains three randomized versions, with EAcc and $\Delta$ rigorously assessing LLMs' true reasoning skills.
• Figure 2: EAcc v.s. $\text{Acc}_v$ on UGMathBench.
• Figure 3: Left: EAcc v.s. Model Size. Right: 2-RE v.s. Model Size. The comparison chart of performance versus performance (EAcc and RE) on UGMathBench for all LLMs evaluated, with models from the same series connected by lines of the same color. The horizontal dotted lines represent the score of close-source LLMs.
• Figure 4: Relationship between EAcc, subject, and level of difficulty. (a) EAcc of different subjects, averaged across all models. Each bar consists of several segments with colors indicating their corresponding difficulty level. Notice that the length of each color segment only indicates its proportion within all problems of all levels within that subject, and is not comparable between levels or subjects. (b) EAcc of different levels, averaged across all subjects. Only models with top-10 EAcc are included for brevity. Levels 5 and 6 are combined since level 6 has few samples.
• Figure 5: Error Analysis of OpenAI-o1-mini on UGMathBench.