Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math

Dingjie Song, Tianlong Xu, Yi-Fan Zhang, Hang Li, Zhiling Yan, Xing Fan, Haoyang Li, Lichao Sun, Qingsong Wen

Abstract

Assessing student handwritten scratchwork is crucial for personalized educational feedback but presents unique challenges due to diverse handwriting, complex layouts, and varied problem-solving approaches. Existing educational NLP primarily focuses on textual responses and neglects the complexity and multimodality inherent in authentic handwritten scratchwork. Current multimodal large language models (MLLMs) excel at visual reasoning but typically adopt an "examinee perspective", prioritizing generating correct answers rather than diagnosing student errors. To bridge these gaps, we introduce ScratchMath, a novel benchmark specifically designed for explaining and classifying errors in authentic handwritten mathematics scratchwork. Our dataset comprises 1,720 mathematics samples from Chinese primary and middle school students, supporting two key tasks: Error Cause Explanation (ECE) and Error Cause Classification (ECC), with seven defined error types. The dataset is meticulously annotated through rigorous human-machine collaborative approaches involving multiple stages of expert labeling, review, and verification. We systematically evaluate 16 leading MLLMs on ScratchMath, revealing significant performance gaps relative to human experts, especially in visual recognition and logical reasoning. Proprietary models notably outperform open-source models, with large reasoning models showing strong potential for error explanation. All evaluation data and frameworks are publicly available to facilitate further research.

Can MLLMs Read Students' Minds? Unpacking Multimodal Error Analysis in Handwritten Math

Abstract

Assessing student handwritten scratchwork is crucial for personalized educational feedback but presents unique challenges due to diverse handwriting, complex layouts, and varied problem-solving approaches. Existing educational NLP primarily focuses on textual responses and neglects the complexity and multimodality inherent in authentic handwritten scratchwork. Current multimodal large language models (MLLMs) excel at visual reasoning but typically adopt an "examinee perspective", prioritizing generating correct answers rather than diagnosing student errors. To bridge these gaps, we introduce ScratchMath, a novel benchmark specifically designed for explaining and classifying errors in authentic handwritten mathematics scratchwork. Our dataset comprises 1,720 mathematics samples from Chinese primary and middle school students, supporting two key tasks: Error Cause Explanation (ECE) and Error Cause Classification (ECC), with seven defined error types. The dataset is meticulously annotated through rigorous human-machine collaborative approaches involving multiple stages of expert labeling, review, and verification. We systematically evaluate 16 leading MLLMs on ScratchMath, revealing significant performance gaps relative to human experts, especially in visual recognition and logical reasoning. Proprietary models notably outperform open-source models, with large reasoning models showing strong potential for error explanation. All evaluation data and frameworks are publicly available to facilitate further research.

Paper Structure

This paper contains 31 sections, 1 equation, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs and MLLMs for Education
  4. LLMs and MLLMs for Mathematical Reasoning
  5. The ScratchMath Benchmark
  6. Task Definition and Taxonomy.
  7. Formal Task Definition
  8. Error Cause Taxonomy
  9. Evaluation Metrics
  10. Dataset Construction
  11. Part I. Student Data Collection
  12. Part II. Answer Pre-annotation
  13. Part III. Expert Human Annotation
  14. Part IV. Post-annotation Quality Verification
  15. Data Statistics
  16. ...and 16 more sections

Figures (9)

  • Figure 1: Overview of this work. (Top) An example illustrating two tasks proposed. (Bottom) Summary of the three research questions addressed in this study.
  • Figure 1: Dataset statistics for the ScratchMath
  • Figure 2: Overview of ScratchMath (with models' answer and human labeled answer translated in English). It illustrates dataset structure, tasks (ECE and ECC), multimodal model predictions, and expert human annotations.
  • Figure 3: An overview of the ScratchMath benchmark construction pipeline.
  • Figure 4: Distribution of error cause classification labels for primary school and middle school problems
  • ...and 4 more figures