Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SketchJudge: A Diagnostic Benchmark for Grading Hand-drawn Diagrams with Multimodal Large Language Models

Yuhang Su, Mei Wang, Yaoyao Zhong, Guozhang Li, Shixing Li, Yihan Feng, Hua Huang

TL;DR

SketchJudge introduces a diagnostic, four-domain benchmark for grading hand-drawn STEM diagrams, emphasizing perceptual, structural, semantic, and metacognitive reasoning beyond solving. It pairs questions (Q) with clean references (R) and student sketches (A), providing binary correctness labels and fine-grained error taxonomies across geometry, physics, charts, and flowcharts, totaling 1,015 sketches over 300 problems. Evaluations of 16 MLLMs reveal a persistent gap to human performance, with closed-source models generally outperforming open-source ones, and with WithRef aiding correctness more than fine-grained error diagnosis. The work demonstrates the need for deeper diagram-level reasoning and offers a foundation for future research in diagnostic grading, tool-enabled reasoning, and education-focused AI deployment.

Abstract

While Multimodal Large Language Models (MLLMs) have achieved remarkable progress in visual understanding, they often struggle when faced with the unstructured and ambiguous nature of human-generated sketches. This limitation is particularly pronounced in the underexplored task of visual grading, where models should not only solve a problem but also diagnose errors in hand-drawn diagrams. Such diagnostic capabilities depend on complex structural, semantic, and metacognitive reasoning. To bridge this gap, we introduce SketchJudge, a novel benchmark tailored for evaluating MLLMs as graders of hand-drawn STEM diagrams. SketchJudge encompasses 1,015 hand-drawn student responses across four domains: geometry, physics, charts, and flowcharts, featuring diverse stylistic variations and distinct error types. Evaluations on SketchJudge demonstrate that even advanced MLLMs lag significantly behind humans, validating the benchmark's effectiveness in exposing the fragility of current vision-language alignment in symbolic and noisy contexts. All data, code, and evaluation scripts are publicly available at https://github.com/yuhangsu82/SketchJudge.

SketchJudge: A Diagnostic Benchmark for Grading Hand-drawn Diagrams with Multimodal Large Language Models

TL;DR

SketchJudge introduces a diagnostic, four-domain benchmark for grading hand-drawn STEM diagrams, emphasizing perceptual, structural, semantic, and metacognitive reasoning beyond solving. It pairs questions (Q) with clean references (R) and student sketches (A), providing binary correctness labels and fine-grained error taxonomies across geometry, physics, charts, and flowcharts, totaling 1,015 sketches over 300 problems. Evaluations of 16 MLLMs reveal a persistent gap to human performance, with closed-source models generally outperforming open-source ones, and with WithRef aiding correctness more than fine-grained error diagnosis. The work demonstrates the need for deeper diagram-level reasoning and offers a foundation for future research in diagnostic grading, tool-enabled reasoning, and education-focused AI deployment.

Abstract

While Multimodal Large Language Models (MLLMs) have achieved remarkable progress in visual understanding, they often struggle when faced with the unstructured and ambiguous nature of human-generated sketches. This limitation is particularly pronounced in the underexplored task of visual grading, where models should not only solve a problem but also diagnose errors in hand-drawn diagrams. Such diagnostic capabilities depend on complex structural, semantic, and metacognitive reasoning. To bridge this gap, we introduce SketchJudge, a novel benchmark tailored for evaluating MLLMs as graders of hand-drawn STEM diagrams. SketchJudge encompasses 1,015 hand-drawn student responses across four domains: geometry, physics, charts, and flowcharts, featuring diverse stylistic variations and distinct error types. Evaluations on SketchJudge demonstrate that even advanced MLLMs lag significantly behind humans, validating the benchmark's effectiveness in exposing the fragility of current vision-language alignment in symbolic and noisy contexts. All data, code, and evaluation scripts are publicly available at https://github.com/yuhangsu82/SketchJudge.
Paper Structure (67 sections, 12 equations, 23 figures, 26 tables)

This paper contains 67 sections, 12 equations, 23 figures, 26 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Benchmarks for MLLMs.
  4. Evaluation Paradigms: Solver vs. Judge.
  5. Data Realism and Educational Contexts.
  6. Benchmark Design
  7. Overview of SketchJudge
  8. Data Collection
  9. Question.
  10. Reference.
  11. Answer.
  12. Annotation Protocol
  13. Stage 1: Bottom-up taxonomy induction.
  14. Stage 2: Expert refinement and taxonomy-based labeling.
  15. Challenges in Diagram Grading
  16. ...and 52 more sections

Figures (23)

  • Figure 1: Distribution of annotated error types across four task domains in SketchJudge.
  • Figure 2: Representative SketchJudge instances across four domains, showing the question $Q$, reference diagram $R$, and student answers $A$ with correctness labels and error types.
  • Figure 3: Illustration of the diagram grading task in SketchJudge.
  • Figure 4: Strictness and leniency across models.
  • Figure 5: Effect of reference diagrams across models. (Left) Accuracy comparison between WithRef and NoRef settings. (Right) Example-based F1 comparison under the same conditions.
  • ...and 18 more figures