MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark

TL;DR

MathBench introduces a multilingual, five-stage benchmark that jointly evaluates theoretical understanding and practical problem solving in mathematics for LLMs. It deploys a hierarchical knowledge system and dual question streams (theory and application), with quality screening and a 3709-question corpus across Chinese and English. Using CircularEval and Perplexity within the OpenCompass framework, the study benchmarked 20+ models, revealing that larger models, especially GPT-4o, excel in both theory and application, while bilingual performance and application tasks reveal distinct gaps. The work highlights how reasoning strategies (CoT, knowledge infusion) and tools (Code Agent) influence performance and provides a foundation for future improvements in mathematical reasoning and multilingual evaluation.

Abstract

Recent advancements in large language models (LLMs) have showcased significant improvements in mathematics. However, traditional math benchmarks like GSM8k offer a unidimensional perspective, falling short in providing a holistic assessment of the LLMs' math capabilities. To address this gap, we introduce MathBench, a new benchmark that rigorously assesses the mathematical capabilities of large language models. MathBench spans a wide range of mathematical disciplines, offering a detailed evaluation of both theoretical understanding and practical problem-solving skills. The benchmark progresses through five distinct stages, from basic arithmetic to college mathematics, and is structured to evaluate models at various depths of knowledge. Each stage includes theoretical questions and application problems, allowing us to measure a model's mathematical proficiency and its ability to apply concepts in practical scenarios. MathBench aims to enhance the evaluation of LLMs' mathematical abilities, providing a nuanced view of their knowledge understanding levels and problem solving skills in a bilingual context. The project is released at https://github.com/open-compass/MathBench .

Figures (24)

• Figure 1: MathBench Overview. MathBench comprises multiple stages of progressively increasing challenges. Each stage encompasses bilingual theoretical and application-oriented questions, with each question precisely tagged with a three-level label to indicate its fine-grained knowledge point.
• Figure 2: Framework of MathBench, We first categorize the mathematical content into four main educational stages and one basic arithmetic stage. Then, we extend from these to fill in two more fine-grained levels of knowledge points, forming the final MathBench framework.
• Figure 3: Scores of Application Problems at Each Stage. Models exhibit similar performances in Arithmetic and Primary stages, while demonstrating a clear performance decline from Primary to College stages.
• Figure 4: CE Evaluation vs. ACC Evaluation. The ACC evaluation queries the model once per question and checks for correctness, whereas the CE (CircularEval) conducts a more stringent and robust assessment by rolling out evaluations four times with shuffled answer options, deeming a question correct only if all attempts are accurate. The percentages depicted in the figure represent the performance decrease of models in the CE evaluation compared to the ACC evaluation.
• Figure 5: Bilingual Comparison on MathBench. showcasing scores in Chinese, English , and their average for the gray dashed line. The Arithmetic stage is not include because there no impact of language in it.