DrawEduMath: Evaluating Vision Language Models with Expert-Annotated Students' Hand-Drawn Math Images

TL;DR

DrawEduMath introduces a novel benchmark for vision-language models (VLMs) grounded in real-world education: 2,030 images of K-12 handwritten math work with expert teacher descriptions and 11,661 teacher-written QA pairs, augmented by $44{,}362$ synthetic QA pairs generated from descriptions using Claude-3.5 and GPT-4o. The dataset is designed to test mathematical reasoning over noisy, hand-drawn content and to capture pedagogical insights beyond traditional OCR-focused datasets. A seven-category QA taxonomy and extensive model benchmarking (including GPT-4o, Claude, Gemini, and Llama) reveal that current VLMs struggle on handwritten content, with closed models outperforming open ones, though synthetic QA can reproduce some model rankings. The release of DrawEduMath aims to spur progress in educational VLMs, enabling more nuanced evaluation of mathematical reasoning and feedback capabilities in real classroom contexts.

Abstract

In real-world settings, vision language models (VLMs) should robustly handle naturalistic, noisy visual content as well as domain-specific language and concepts. For example, K-12 educators using digital learning platforms may need to examine and provide feedback across many images of students' math work. To assess the potential of VLMs to support educators in settings like this one, we introduce DrawEduMath, an English-language dataset of 2,030 images of students' handwritten responses to K-12 math problems. Teachers provided detailed annotations, including free-form descriptions of each image and 11,661 question-answer (QA) pairs. These annotations capture a wealth of pedagogical insights, ranging from students' problem-solving strategies to the composition of their drawings, diagrams, and writing. We evaluate VLMs on teachers' QA pairs, as well as 44,362 synthetic QA pairs derived from teachers' descriptions using language models (LMs). We show that even state-of-the-art VLMs leave much room for improvement on DrawEduMath questions. We also find that synthetic QAs, though imperfect, can yield similar model rankings as teacher-written QAs. We release DrawEduMath to support the evaluation of VLMs' abilities to reason mathematically over images gathered with educational contexts in mind.

Figures (10)

• Figure 1: Each image in our dataset is a concatenation of a math problem on the left with a student response on the right. Teachers describe the student's response to the problem, and then a model, such as GPT-4o shown here, writes QA pairs extracted from facets of the description. More example images, along with teacher-written QA, are shown in Figure \ref{['fig:example_data_teacher']}.
• Figure 2: For some annotators, their recorded descriptions of images are longer or require less time than typed ones. Annotation length is calculated based on white-spaced-separated tokens.
• Figure 3: Examples of teachers' answers to a question asking about possible errors in students' responses to math problems. All three examples of students' hand-drawn responses are for the same math problem asking students to draw and shade units on fraction strips to show 4 thirds, shown on the left.
• Figure 4: A screenshot of our recording website, where teachers would view an image from our dataset and either write or record a description of the student's response. Typically, "unknown teacher ID" would include the currently annotating teacher's ID.
• Figure 5: A screenshot of the interface teachers used to write answers to teacher-written questions about students' responses. Typically, "unknown teacher ID" would include the currently annotating teacher's ID.