Investigating Large Language Models in Diagnosing Students' Cognitive Skills in Math Problem-solving

TL;DR

The paper attacks the problem of diagnosing students' cognitive skills in math using automated LLM-based assessment by introducing MathCog, a TIMSS-aligned, multimodal benchmark with expert diagnoses. It evaluates 16 LLMs under text-only and multimodal inputs with Chain-of-Thought prompting, finding that no model achieves reliable diagnostic performance (all $F1<0.5$) and that overconfidence is common, though model size positively correlates with accuracy (e.g., $r_s=0.771$). The results highlight the current limitations of LLMs for high-stakes cognitive-diagnosis in math and point to the need for improved prompting, multimodal representations, and calibration of model confidence. The work provides a foundation for future, more robust automatic cognitive-diagnosis systems in math education and outlines concrete directions for dataset expansion and methodological refinements.

Abstract

Mathematics learning entails mastery of both content knowledge and cognitive processing of knowing, applying, and reasoning with it. Automated math assessment primarily has focused on grading students' exhibition of content knowledge by finding textual evidence, such as specific numbers, formulas, and statements. Recent advancements in problem-solving, image recognition, and reasoning capabilities of large language models (LLMs) show promise for nuanced evaluation of students' cognitive skills. Diagnosing cognitive skills needs to infer students' thinking processes beyond textual evidence, which is an underexplored task in LLM-based automated assessment. In this work, we investigate how state-of-the-art LLMs diagnose students' cognitive skills in mathematics. We constructed MathCog, a novel benchmark dataset comprising 639 student responses to 110 expert-curated middle school math problems, each annotated with detailed teachers' diagnoses based on cognitive skill checklists. Using MathCog, we evaluated 16 closed and open LLMs of varying model sizes and vendors. Our evaluation reveals that even the state-of-the-art LLMs struggle with the task, all F1 scores below 0.5, and tend to exhibit strong false confidence for incorrect cases ($r_s=.617$). We also found that model size positively correlates with the diagnosis performance ($r_s=.771$). Finally, we discuss the implications of these findings, the overconfidence issue, and directions for improving automated cognitive skill diagnosis.

Figures (5)

• Figure 1: The dataset creation process and the number of problem types, problems, and student responses at each filtering stage.
• Figure 2: The performance of five state-of-the-art LLMs across cognitive skill types. The left plot shows the minimum (bottom $\times$), maximum (top $\times$), and average (middle •) F1 scores for each skill category, while the right plot presents the corresponding accuracy metrics.
• Figure 3: Impact of multimodal input (left) and reasoning capability (right) on LLMs' performance in cognitive skill diagnosis. On the left, each adjacent pair of bars shows performance with and without the image input from the same model family. On the right, models with reasoning capabilities are grouped at the front.
• Figure 4: Impact of model size on LLM performance in cognitive skill diagnosis. Models are grouped into large, medium, and small categories.
• Figure 5: Illustrative examples of diagnosis check items and student responses that LLMs failed to diagnose correctly. Evidence for human judgment is marked with a red box.