Can Vision-Language Models Evaluate Handwritten Math?

TL;DR

Can Vision-Language Models Evaluate Handwritten Math? introduces Fermat, a large, open-source benchmark with 2,244 perturbed handwritten solutions from 609 grade 7–12 problems to evaluate VLMs on detecting, localizing, and correcting errors in handwritten math. The benchmark employs a four-axis perturbation taxonomy and a human-in-the-loop perturbation process, with handwritten transcription and verification, across three tasks and nine VLMs. Results show substantial gaps in current VLMs’ ability to reason over handwritten content, with the best correction rate around 77% and OCR-based enhancements offering model-dependent gains; a cascaded setup often reduces performance due to early error filtering. An LLM-based evaluator (GPT-4o) provides scalable, reliable assessment aligned with human judgments, and the work is released open-source to spur further research in multimodal handwritten-math evaluation.

Abstract

Recent advancements in Vision-Language Models (VLMs) have opened new possibilities in automatic grading of handwritten student responses, particularly in mathematics. However, a comprehensive study to test the ability of VLMs to evaluate and reason over handwritten content remains absent. To address this gap, we introduce FERMAT, a benchmark designed to assess the ability of VLMs to detect, localize and correct errors in handwritten mathematical content. FERMAT spans four key error dimensions - computational, conceptual, notational, and presentation - and comprises over 2,200 handwritten math solutions derived from 609 manually curated problems from grades 7-12 with intentionally introduced perturbations. Using FERMAT we benchmark nine VLMs across three tasks: error detection, localization, and correction. Our results reveal significant shortcomings in current VLMs in reasoning over handwritten text, with Gemini-1.5-Pro achieving the highest error correction rate (77%). We also observed that some models struggle with processing handwritten content, as their accuracy improves when handwritten inputs are replaced with printed text or images. These findings highlight the limitations of current VLMs and reveal new avenues for improvement. We release FERMAT and all the associated resources in the open-source to drive further research.

Figures (36)

• Figure 1: We introduce Fermat, a novel multimodal benchmark to evaluate VLMs on their ability to detect, reason about, and assess the correctness of handwritten grade-school level math solutions.
• Figure 2: The construction of Fermat involves four steps: (1) sampling problems with detailed solutions from math domains (§\ref{['problem_set_collection']}), (2) defining a perturbation taxonomy (§\ref{['perturbation_taxonomy']}), (3) applying perturbations to solutions (§\ref{['perturbed_set_curation']}), and (4) transcribing the perturbed QA pairs (§\ref{['hw_transcription']}).
• Figure 3: Cascaded black-box evaluation setup, as described in §\ref{['cascaded']}.$GT$ denotes Ground Truth. The total number of correctly evaluated Fermat samples in this setup is represented by the summation of $A0$, $B0$, $C0$, and $D$.
• Figure 4: Performance of VLMs on the error localization task across various benchmark settings. Higher scores ($\uparrow$) indicate better performance.
• Figure 5: Distribution of different error types (left) across educational levels (middle) and math topics (right) within Fermat.