Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MathScape: Benchmarking Multimodal Large Language Models in Real-World Mathematical Contexts

Hao Liang, Linzhuang Sun, Minxuan Zhou, Zirong Chen, Meiyi Qiang, Mingan Lin, Tianpeng Li, Fan Yang, Zenan Zhou, Wentao Zhang

TL;DR

MathScape introduces a real-world multimodal math benchmark with 1,369 problems paired with human-captured images to evaluate multimodal LLMs beyond synthetic data. It implements a three-step data construction pipeline, a two-step long-form evaluation with sub-answer scoring, and a multidimensional taxonomy across question types, knowledge points, and educational levels, all validated by humans. Experiments across nine closed-source and numerous open-source LMMs reveal that state-of-the-art models struggle on real-world image-based math tasks, despite strong PDF-based performance, highlighting gaps in robustness and generalization. The work demonstrates the necessity of real-world benchmarks for advancing multimodal mathematical reasoning and outlines directions for more robust data, evaluation methods, and future research.

Abstract

With the rapid progress of Multimodal LLMs, evaluating their mathematical reasoning capabilities has become an increasingly important research direction. In particular, visual-textual mathematical reasoning serves as a key indicator of an MLLM's ability to comprehend and solve complex, multi-step quantitative problems. While existing benchmarks such as MathVista and MathVerse have advanced the evaluation of multimodal math proficiency, they primarily rely on digitally rendered content and fall short in capturing the complexity of real-world scenarios. To bridge this gap, we introduce MathScape, a novel benchmark focused on assessing MLLMs' reasoning ability in realistic mathematical contexts. MathScape comprises 1,369 high-quality math problems paired with human-captured real-world images, closely reflecting the challenges encountered in practical educational settings. We conduct a thorough multi-dimensional evaluation across nine leading closed-source MLLMs, three open-source MLLMs with over 20 billion parameters, and seven smaller-scale MLLMs. Our results show that even SOTA models struggle with real-world math tasks, lagging behind human performance -- highlighting critical limitations in current model capabilities. Moreover, we find that strong performance on synthetic or digitally rendered images does not guarantee similar effectiveness on real-world tasks. This underscores the necessity of MathScape in the next stage of multimodal mathematical reasoning.

MathScape: Benchmarking Multimodal Large Language Models in Real-World Mathematical Contexts

TL;DR

MathScape introduces a real-world multimodal math benchmark with 1,369 problems paired with human-captured images to evaluate multimodal LLMs beyond synthetic data. It implements a three-step data construction pipeline, a two-step long-form evaluation with sub-answer scoring, and a multidimensional taxonomy across question types, knowledge points, and educational levels, all validated by humans. Experiments across nine closed-source and numerous open-source LMMs reveal that state-of-the-art models struggle on real-world image-based math tasks, despite strong PDF-based performance, highlighting gaps in robustness and generalization. The work demonstrates the necessity of real-world benchmarks for advancing multimodal mathematical reasoning and outlines directions for more robust data, evaluation methods, and future research.

Abstract

With the rapid progress of Multimodal LLMs, evaluating their mathematical reasoning capabilities has become an increasingly important research direction. In particular, visual-textual mathematical reasoning serves as a key indicator of an MLLM's ability to comprehend and solve complex, multi-step quantitative problems. While existing benchmarks such as MathVista and MathVerse have advanced the evaluation of multimodal math proficiency, they primarily rely on digitally rendered content and fall short in capturing the complexity of real-world scenarios. To bridge this gap, we introduce MathScape, a novel benchmark focused on assessing MLLMs' reasoning ability in realistic mathematical contexts. MathScape comprises 1,369 high-quality math problems paired with human-captured real-world images, closely reflecting the challenges encountered in practical educational settings. We conduct a thorough multi-dimensional evaluation across nine leading closed-source MLLMs, three open-source MLLMs with over 20 billion parameters, and seven smaller-scale MLLMs. Our results show that even SOTA models struggle with real-world math tasks, lagging behind human performance -- highlighting critical limitations in current model capabilities. Moreover, we find that strong performance on synthetic or digitally rendered images does not guarantee similar effectiveness on real-world tasks. This underscores the necessity of MathScape in the next stage of multimodal mathematical reasoning.
Paper Structure (15 sections, 4 figures, 2 tables)

This paper contains 15 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmarks for Mathematical Evaluation
  4. Methodology
  5. Construction of MathScape
  6. Multidimensional Evaluation
  7. Evaluation Method
  8. Statistical Characteristics of MathScape
  9. Experiments and Analysis
  10. Experimental Setups
  11. Performance of Various Models
  12. The Importance of Real-World Images in Benchmarking
  13. Stability Results and Analysis
  14. Conclusion
  15. Acknowledgments

Figures (4)

  • Figure 1: MathScape offers a comprehensive collection of math problems from primary school to high school. The problems range in difficulty from easy to difficult, catering to various levels of evaluation.
  • Figure 2: MathScape Data Illustration. We select real-world mathematics examples from various mathematical domains, including geometry, probability and statistics, functions, algebra, and equations.
  • Figure 3: The distribution of MathScape. In (a), we show the proportion based on knowledge areas, while in (b), we present the proportion based on question types.
  • Figure 4: Each problem is tested five times, with the numbers 1 to 5 indicating the number of correct responses. Our results show that only about 25% of the questions are answered correctly in all five attempts.