Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models

Zhong-Zhi Li, Ming-Liang Zhang, Fei Yin, Zhi-Long Ji, Jin-Feng Bai, Zhen-Ru Pan, Fan-Hu Zeng, Jian Xu, Jia-Xin Zhang, Cheng-Lin Liu

TL;DR

This work introduces CMMaTH, the largest Chinese multimodal math benchmark for K-12, to address the scarcity of detailed Chinese multimodal evaluation data. It provides 23856 questions across 13 visual categories and 5 difficulty levels, with thorough knowledge-point annotations and a bilingual English version, plus a lightweight open-source evaluator, GradeGPT, built around a Cross-Lingual-Judge-of-Chain framework. Extensive experiments across open-source and commercial LLMs/LMMs reveal significant gaps in current multimodal Chinese math reasoning and illustrate the challenges of multilingual prompting and diagram-based reasoning. The dataset is designed to be dynamically maintained, enabling rapid, cost-effective benchmarking and iterative development of educational AI for Chinese K-12 contexts.

Abstract

Due to the rapid advancements in multimodal large language models, evaluating their multimodal mathematical capabilities continues to receive wide attention. Despite the datasets like MathVista proposed benchmarks for assessing mathematical capabilities in multimodal scenarios, there is still a lack of corresponding evaluation tools and datasets for fine-grained assessment in the context of K12 education in Chinese language. To systematically evaluate the capability of multimodal large models in solving Chinese multimodal mathematical problems, we propose a Chinese Multi-modal Math Skill Evaluation Benchmark, named CMMaTH, contraining 23k multimodal K12 math related questions, forming the largest Chinese multimodal mathematical problem benchmark to date. CMMaTH questions from elementary to high school levels, provide increased diversity in problem types, solution objectives, visual elements, detailed knowledge points, and standard solution annotations. We have constructed an open-source tool GradeGPT integrated with the CMMaTH dataset, facilitating stable, rapid, and cost-free model evaluation. Our data and code are available.

CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models

TL;DR

This work introduces CMMaTH, the largest Chinese multimodal math benchmark for K-12, to address the scarcity of detailed Chinese multimodal evaluation data. It provides 23856 questions across 13 visual categories and 5 difficulty levels, with thorough knowledge-point annotations and a bilingual English version, plus a lightweight open-source evaluator, GradeGPT, built around a Cross-Lingual-Judge-of-Chain framework. Extensive experiments across open-source and commercial LLMs/LMMs reveal significant gaps in current multimodal Chinese math reasoning and illustrate the challenges of multilingual prompting and diagram-based reasoning. The dataset is designed to be dynamically maintained, enabling rapid, cost-effective benchmarking and iterative development of educational AI for Chinese K-12 contexts.

Abstract

Due to the rapid advancements in multimodal large language models, evaluating their multimodal mathematical capabilities continues to receive wide attention. Despite the datasets like MathVista proposed benchmarks for assessing mathematical capabilities in multimodal scenarios, there is still a lack of corresponding evaluation tools and datasets for fine-grained assessment in the context of K12 education in Chinese language. To systematically evaluate the capability of multimodal large models in solving Chinese multimodal mathematical problems, we propose a Chinese Multi-modal Math Skill Evaluation Benchmark, named CMMaTH, contraining 23k multimodal K12 math related questions, forming the largest Chinese multimodal mathematical problem benchmark to date. CMMaTH questions from elementary to high school levels, provide increased diversity in problem types, solution objectives, visual elements, detailed knowledge points, and standard solution annotations. We have constructed an open-source tool GradeGPT integrated with the CMMaTH dataset, facilitating stable, rapid, and cost-free model evaluation. Our data and code are available.
Paper Structure (34 sections, 2 equations, 23 figures, 13 tables)

This paper contains 34 sections, 2 equations, 23 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Assessment of mathematical abilities
  4. Large Model Evaluation Tool
  5. CMMaTH Dataset
  6. Overview of CMMaTH
  7. Collectioin Guidelines
  8. Data Collections
  9. Comparison with Existing Benchmarks
  10. GradeGPT
  11. Experiments
  12. Main Experiments on LLM/LMMs
  13. Experiments of Cross-language Reason Technology
  14. Error Analysis
  15. Conclusions
  16. ...and 19 more sections

Figures (23)

  • Figure 1: The results of mainstream multimodal large models and pure text large models on the CMMaTH dataset. Left: represents the performance evaluation of selected LMMs and LLMs across various Visual Subjects. Right: the performance assessment of these models on different educational grade-level questions.
  • Figure 1: Key statistics of CMMaTH. The unit of question length is words.
  • Figure 2: Some of the knowledge points involved in the CMMaTH dataset.
  • Figure 3: Instruction Construction Pipeline of GradeGPT
  • Figure 4: Accuracy of LMMs across different types of problems in CMMaTH Benchmark.
  • ...and 18 more figures