ConceptMath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Models

TL;DR

ConceptMath introduces a bilingual, concept-wise benchmark for mathematical reasoning that organizes problems into a three-level hierarchy across four language-system pairs, totaling 214 concepts and 4011 problems. It reveals that many LLMs with strong average performance on traditional benchmarks exhibit large concept-level variability and even failures on basic concepts, with Chinese-language results generally lagging behind English. The paper also proposes an efficient fine-tuning strategy that uses a concept classifier to curate concept-specific data and combine it with existing math datasets, yielding improvements on weaker concepts while preserving broader abilities. Together, ConceptMath provides a fine-grained diagnostic for model weaknesses and a practical pathway to targeted improvements, aiming to guide developers in strengthening foundational mathematical reasoning in multilingual settings.

Abstract

This paper introduces ConceptMath, a bilingual (English and Chinese), fine-grained benchmark that evaluates concept-wise mathematical reasoning of Large Language Models (LLMs). Unlike traditional benchmarks that evaluate general mathematical reasoning with an average accuracy, ConceptMath systematically organizes math problems under a hierarchy of math concepts, so that mathematical reasoning can be evaluated at different granularity with concept-wise accuracies. Based on our ConcepthMath, we evaluate a broad range of LLMs, and we observe existing LLMs, though achieving high average accuracies on traditional benchmarks, exhibit significant performance variations across different math concepts and may even fail catastrophically on the most basic ones. Besides, we also introduce an efficient fine-tuning strategy to enhance the weaknesses of existing LLMs. Finally, we hope ConceptMath could guide the developers to understand the fine-grained mathematical abilities of their models and facilitate the growth of foundation models.

Figures (17)

• Figure 1: The concept-wise accuracies of LLaMA2-13B and the fine-tuned version based on our efficient fine-tuning method (i.e., LLaMA2-FT).
• Figure 2: Diagram overview of four concept systems in ConceptMath. We have provided translated Chinese concept names in English (See Appendix \ref{['app: concept']}).
• Figure 3: Length distributions of our ConceptMath.
• Figure 4: Mean accuracies for English, Chinese, and overall educational systems.
• Figure 5: Mean concept accuracies on Middle-EN.