EDU-CIRCUIT-HW: Evaluating Multimodal Large Language Models on Real-World University-Level STEM Student Handwritten Solutions

TL;DR

This work presents EDU-CIRCUIT-HW, a real-world benchmark of 1,300+ authentic university-level handwritten STEM solutions to probe multimodal large language models (MLLMs) in both visual recognition and downstream auto-grading. It introduces a diagnostic pipeline that jointly evaluates upstream handwritten solution recognition fidelity and downstream grading quality, revealing a substantial latent error rate that traditional task-focused assessments miss. The authors validate an LLM-based recognition-error detector and a four-category error taxonomy, showing that better visual understanding reduces error propagation to grading, yet still lags behind human graders on fine-grained judgments. A case study demonstrates a human-in-the-loop regrading module that leverages detected error patterns to suppress recognition errors with minimal human effort, significantly boosting robustness on unseen solutions. These findings underscore the need for trustworthy AI-enabled educational tools and provide a practical blueprint for improving reliability in real-world educational deployments.

Abstract

Multimodal Large Language Models (MLLMs) hold significant promise for revolutionizing traditional education and reducing teachers' workload. However, accurately interpreting unconstrained STEM student handwritten solutions with intertwined mathematical formulas, diagrams, and textual reasoning poses a significant challenge due to the lack of authentic and domain-specific benchmarks. Additionally, current evaluation paradigms predominantly rely on the outcomes of downstream tasks (e.g., auto-grading), which often probe only a subset of the recognized content, thereby failing to capture the MLLMs' understanding of complex handwritten logic as a whole. To bridge this gap, we release EDU-CIRCUIT-HW, a dataset consisting of 1,300+ authentic student handwritten solutions from a university-level STEM course. Utilizing the expert-verified verbatim transcriptions and grading reports of student solutions, we simultaneously evaluate various MLLMs' upstream recognition fidelity and downstream auto-grading performance. Our evaluation uncovers an astonishing scale of latent failures within MLLM-recognized student handwritten content, highlighting the models' insufficient reliability for auto-grading and other understanding-oriented applications in high-stakes educational settings. In solution, we present a case study demonstrating that leveraging identified error patterns to preemptively detect and rectify recognition errors, with only minimal human intervention (approximately 4% of the total solutions), can significantly enhance the robustness of the deployed AI-enabled grading system on unseen student solutions.

Figures (24)

• Figure 1: An exemplary STEM student handwritten solution auto-grading workflow: MLLM (e.g., Gemini-2.5-Pro) recognizes ①, ②, ③, and other visual contents of the solution, which serve as inputs for the consequent AI grading. Notably, recognition errors (① and ②) may not influence the grading if they fall outside the specific grading criteria. As a result, this successful grading masks underlying recognition failures, leading users to overestimate the MLLM's visual understanding performance on the student solution.
• Figure 2: The demonstration on our recognition error detection. Given a handwritten solution, the MLLM's recognition result is compared with the expert-verified transcriptions in a full-text level, after which all discrepant items within the recognition result are listed as recognition errors. Note that we regard two items as aligned when they are semantically equivalent in engineering education (e.g., KCL: "out" and KCL: @ "out" are regarded as semantically aligned despite the additional "@").
• Figure 3: The proposed auto-grading pipeline. The "vanilla grading pipeline" in green box is a widely-used auto-grading paradigm in AI-enabled education field. In our experiment (Section \ref{['sec: exp_recognition']}), we detect and analyze recognition errors within "MLLM Recognized Text" utilizing the automated pipeline introduced in Section \ref{['sec: Recognizing the Handwritten Content']}, and further evaluate "Grading Report" by benchmarking it against the "Expert's Grading Report" in yellow box. In pink box, we present an exemplary regrading module where we summarize the detected recognition error patterns and leverage it to filter out potential recognition errors within "MLLM Recognized Text", and let LLM/human regraders to reassess those solutions flagged with containing potential recognition error, more details can be found in Section \ref{['sec: case_study']}.
• Figure 4: Comparisons of different MLLMs' recognition error counts over 4 error categories (defined in Table \ref{['tab:error_taxonomy']}) and the corresponding error impact rate (EIR) to the downstream LLM grading performance in Table \ref{['tab: exp_result1']}. The annotation above each bar presents the error count (line 1) and the EIR (line 2 in gray) of this model in the category.
• Figure 5: Prompt used to categorize recognition errors according to the taxonomy defined in Table \ref{['tab:error_taxonomy']}, where the four categories correspond to "Symbol & Character", "Structural & Notational", "Diagrammatic", and "Textual & Logical" errors.