Tracing Mathematical Proficiency Through Problem-Solving Processes

TL;DR

This work addresses explainability gaps in knowledge tracing by incorporating students' problem-solving processes into KT (KT-PSP). It introduces KT-PSP-25, a math dataset with OCR-transcribed PSPs, and StatusKT, a three-stage LLM pipeline that derives mathematical proficiency indicators and MP ratios to augment KT models. Across multiple baselines, StatusKT improves prediction accuracy (ACC and AUC) and provides interpretable proficiency signals, demonstrating the value of process-level reasoning for pedagogy. The approach highlights the importance of integrating reasoning traces into student models to achieve more accurate, interpretable, and practically useful KT systems.

Abstract

Knowledge Tracing (KT) aims to model student's knowledge state and predict future performance to enable personalized learning in Intelligent Tutoring Systems. However, traditional KT methods face fundamental limitations in explainability, as they rely solely on the response correctness, neglecting the rich information embedded in students' problem-solving processes. To address this gap, we propose Knowledge Tracing Leveraging Problem-Solving Process (KT-PSP), which incorporates students' problem-solving processes to capture the multidimensional aspects of mathematical proficiency. We also introduce KT-PSP-25, a new dataset specifically designed for the KT-PSP. Building on this, we present StatusKT, a KT framework that employs a teacher-student-teacher three-stage LLM pipeline to extract students' MP as intermediate signals. In this pipeline, the teacher LLM first extracts problem-specific proficiency indicators, then a student LLM generates responses based on the student's solution process, and a teacher LLM evaluates these responses to determine mastery of each indicator. The experimental results on KT-PSP-25 demonstrate that StatusKT improves the prediction performance of existing KT methods. Moreover, StatusKT provides interpretable explanations for its predictions by explicitly modeling students' mathematical proficiency.

Figures (7)

• Figure 1: (a) Conventional KT utilizes limited metadata, lacking interpretability. (b) By incorporating problems, students reasoning, and teacher comments, StatusKT captures granular proficiency like a teacher.
• Figure 2: StatusKT Framework Overview. (a) Students' problem-solving processes in math tasks provide rich cues about their understanding, which our framework leverages to derive mathematical proficiency (MP) indicators. (b) A teacher LLM analyzes each problem's text and knowledge components to construct MP indicators, while a student LLM identifies which it understands and can explain. Using these responses, the teacher LLM integrates the results to generate an MP ratio quantifying each student's mathematical proficiency. (c) Using the MP ratio, the KT model predicts both students' future performance and problem-level MP ratios, offering a strong rationale for proficiency assessment.
• Figure 3: Prompt used for GPT-based OCR to convert students' handwritten problem-solving processes into text.
• Figure 4: Prompt used for refining the GPT-based OCR outputs to improve the accuracy and consistency of transcribed student solutions in our KT-PSP-25.
• Figure 5: Prompt used for extracting the MP indicators from the given problem in StatusKT. Prompt inputs are boldfaced.