VEHME: A Vision-Language Model For Evaluating Handwritten Mathematics Expressions

TL;DR

VEHME tackles open-form handwritten mathematics grading by casting the task as a vision-language evaluation problem. It combines a dual-phase training regime—supervised fine-tuning on reasoning-rich syntheses from $QwQ{-}32B$ and reinforcement learning via Group Relative Policy Optimization—with an Expression-Aware Visual Prompting module to robustly localize errors in diverse layouts. The approach achieves state-of-the-art performance among open-source models on AIHub and FERMAT, approaching proprietary systems, and demonstrates strong robustness to rotation and layout variability. This work provides an accessible, scalable framework for automated math assessment and highlights the value of spatial prompts and structured reasoning in multimodal evaluation tasks.

Abstract

Automatically assessing handwritten mathematical solutions is an important problem in educational technology with practical applications, but it remains a significant challenge due to the diverse formats, unstructured layouts, and symbolic complexity of student work. To address this challenge, we introduce VEHME-a Vision-Language Model for Evaluating Handwritten Mathematics Expressions-designed to assess open-form handwritten math responses with high accuracy and interpretable reasoning traces. VEHME integrates a two-phase training pipeline: (i) supervised fine-tuning using structured reasoning data, and (ii) reinforcement learning that aligns model outputs with multi-dimensional grading objectives, including correctness, reasoning depth, and error localization. To enhance spatial understanding, we propose an Expression-Aware Visual Prompting Module, trained on our synthesized multi-line math expressions dataset to robustly guide attention in visually heterogeneous inputs. Evaluated on AIHub and FERMAT datasets, VEHME achieves state-of-the-art performance among open-source models and approaches the accuracy of proprietary systems, demonstrating its potential as a scalable and accessible tool for automated math assessment. Our training and experiment code is publicly available at our GitHub repository.

Figures (14)

• Figure 1: Our model takes a question, reference answer, and student answer image as input to predict the correctness of the student's solution and identify any error locations, if the solution is incorrect.
• Figure 2: VEHME overview. Initially, a VLM is fine-tuned to generate outputs in the desired format using synthesized data from Sec \ref{['sec:data-synthesis']}. A subsequent Preference Optimization step trains the model using GRPO deepseekmath method. This optimization is guided by rewards described in Sec \ref{['sec:reward-modeling']} and takes the given problem, reference answer, and an expression-aware visual-prompted image from the student's answer as input.
• Figure 3: Process for synthesizing SFT data. QwQ-32B creates structured feedback from inputs (q, r, s) under a token budget M. Truncated outputs are repaired using a output processing module (Appendix \ref{['sec:data-syn']}) with grammar-constrained decoding (Appendix \ref{['sec:data-syn']}) to ensure valid SFT data.
• Figure 4: EVPM training pipeline. The model learns to predict ground truth bounding boxes for augmented mathematical expressions which are drawn onto a white backgrounds after augmentation.
• Figure 5: VLM evaluation procedure for student solutions, encompassing Error Detection (ED) and Error Localization (EL). ED serves to identify the presence of errors, while EL is responsible for determining the specific location of any detected error.