Grading Handwritten Engineering Exams with Multimodal Large Language Models

TL;DR

This paper tackles the scalability of grading handwritten STEM exams by introducing an end-to-end pipeline that uses multimodal LLMs while preserving standard exam formats. The method grounds grading on a handwritten reference solution that is converted to a text summary, and employs a multi-stage design with presence checks, an ensemble of $K=3$ graders, supervisor aggregation, and deterministic post-processing to yield auditable reports. In a held-out Slovenian quiz with hand-drawn diagrams, state-of-the-art backends achieve close agreement with lecturer grades and maintain a manageable manual-review rate around $D_{max}=40$, with notable improvements over trivial prompting and without reference grounding. Ablation studies confirm that structured prompting and reference grounding are essential for accuracy, and the work highlights the practical potential and necessary safeguards for deploying automated grading in real courses. The authors also discuss data privacy, provide a clean-room evaluation protocol, and plan to broaden validation and release code and datasets for standardized evaluation.

Abstract

Handwritten STEM exams capture open-ended reasoning and diagrams, but manual grading is slow and difficult to scale. We present an end-to-end workflow for grading scanned handwritten engineering quizzes with multimodal large language models (LLMs) that preserves the standard exam process (A4 paper, unconstrained student handwriting). The lecturer provides only a handwritten reference solution (100%) and a short set of grading rules; the reference is converted into a text-only summary that conditions grading without exposing the reference scan. Reliability is achieved through a multi-stage design with a format/presence check to prevent grading blank answers, an ensemble of independent graders, supervisor aggregation, and rigid templates with deterministic validation to produce auditable, machine-parseable reports. We evaluate the frozen pipeline in a clean-room protocol on a held-out real course quiz in Slovenian, including hand-drawn circuit schematics. With state-of-the-art backends (GPT-5.2 and Gemini-3 Pro), the full pipeline achieves $\approx$8-point mean absolute difference to lecturer grades with low bias and an estimated manual-review trigger rate of $\approx$17% at $D_{\max}=40$. Ablations show that trivial prompting and removing the reference solution substantially degrade accuracy and introduce systematic over-grading, confirming that structured prompting and reference grounding are essential.

Figures (5)

• Figure 1: Sample handwritten answer with diagrams (e.g. circuits) that our system is designed to handle. Sample provided by authors (not student work). The link to the full exam and its grading result are provided as supplementary material in the last section of the paper.
• Figure 2: System overview. Green boxes denote data artifacts, red ellipses denote prompt pairs and templates, and blue boxes denote processing stages. All blue processing stages invoke a multimodal LLM backend (e.g., GPT-4o, GPT-5.x, Gemini, or Mistral), while reference solution scans are converted into a text-only summary that is injected into grading prompts.
• Figure 3: Baseline exam-level performance across backends (mean over $R=3$ repetitions where available; GPT-5.2-pro: single run).
• Figure 4: Full pipeline vs. trivial prompting for two strong backends (trivial: no rules, no reference, no supervisor).
• Figure 5: Estimated manual review trigger rate vs. disagreement threshold $D_{\max}$ (mean over repetitions where available).