Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Training-Free Large Reasoning Model-based Knowledge Tracing Framework for Unified Prediction and Prescription

Unggi Lee, Joo Young Kim, Ran Ju, Minyoung Jung, Jeyeon Eo

TL;DR

This work introduces Thinking-KT, a training-free knowledge tracing framework that leverages test-time scaling (TTS) to empower small LLMs to predict learner mastery while jointly generating personalized feedback and learning recommendations in a single inference. By employing structured prompts and a controlled Thinking Budget, the approach produces explicit reasoning traces that improve prediction accuracy (AUC) and pedagogical quality without fine-tuning. Across multiple KT benchmarks, Thinking-KT demonstrates robust KT performance with unified outputs that do not compromise accuracy, while revealing how reasoning depth and trace structure correlate with diagnostic effectiveness. The framework offers a practical, scalable path toward unified intelligent tutoring systems that are interpretable and responsive in real time.

Abstract

Knowledge Tracing (KT) aims to estimate a learner's evolving mastery based on interaction histories. Recent studies have explored Large Language Models (LLMs) for KT via autoregressive nature, but such approaches typically require fine-tuning and exhibit unstable or near-random performance. Moreover, prior KT systems primarily focus on prediction and rely on multi-stage pipelines for feedback and recommendation, resulting in increased system complexity and resources. To address this gap, we propose Thinking-KT, a training-free KT framework that incorporates Test-Time Scaling (TTS), enabling even small LLMs to achieve competitive KT performance. Moreover, in this framework, a small LLM can jointly perform KT prediction, personalized feedback generation, and learning recommendation in a unified output without degrading prediction accuracy. Beyond performance, we present the systematic analysis of reasoning traces in KT. Our results demonstrate that TTS is a critical yet underexplored factor in LLM-based KT, and that small LLMs can serve as unified ITS engines.

A Training-Free Large Reasoning Model-based Knowledge Tracing Framework for Unified Prediction and Prescription

TL;DR

This work introduces Thinking-KT, a training-free knowledge tracing framework that leverages test-time scaling (TTS) to empower small LLMs to predict learner mastery while jointly generating personalized feedback and learning recommendations in a single inference. By employing structured prompts and a controlled Thinking Budget, the approach produces explicit reasoning traces that improve prediction accuracy (AUC) and pedagogical quality without fine-tuning. Across multiple KT benchmarks, Thinking-KT demonstrates robust KT performance with unified outputs that do not compromise accuracy, while revealing how reasoning depth and trace structure correlate with diagnostic effectiveness. The framework offers a practical, scalable path toward unified intelligent tutoring systems that are interpretable and responsive in real time.

Abstract

Knowledge Tracing (KT) aims to estimate a learner's evolving mastery based on interaction histories. Recent studies have explored Large Language Models (LLMs) for KT via autoregressive nature, but such approaches typically require fine-tuning and exhibit unstable or near-random performance. Moreover, prior KT systems primarily focus on prediction and rely on multi-stage pipelines for feedback and recommendation, resulting in increased system complexity and resources. To address this gap, we propose Thinking-KT, a training-free KT framework that incorporates Test-Time Scaling (TTS), enabling even small LLMs to achieve competitive KT performance. Moreover, in this framework, a small LLM can jointly perform KT prediction, personalized feedback generation, and learning recommendation in a unified output without degrading prediction accuracy. Beyond performance, we present the systematic analysis of reasoning traces in KT. Our results demonstrate that TTS is a critical yet underexplored factor in LLM-based KT, and that small LLMs can serve as unified ITS engines.
Paper Structure (33 sections, 5 equations, 4 figures, 7 tables)

This paper contains 33 sections, 5 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Problem Definition
  4. Thinking-KT
  5. Structured Prompt Setting
  6. TTS
  7. Unified Multi-Task Output
  8. Final Formulation
  9. Evaluation
  10. Evaluation for KT prediction
  11. Evaluation for FB and Rec
  12. Results
  13. Experiment Settings
  14. KT Performance
  15. Unified Output Performance
  16. ...and 18 more sections

Figures (4)

  • Figure 1: Comparing No-Think and Thinking-KT across KT performance and pedagogical quality of feedback and recommendations. TTS consistently improves both prediction accuracy and pedagogical quality without training.
  • Figure 2: Overall framework of Thinking-KT. Given a learner’s problem-solving history, the LLM receives a structured prompt and performs TTS before prediction. A single inference produces (1) KT prediction, (2) personalized diagnostic FB, and (3) next-step learning Rec. The right panel illustrates the evaluation pipeline for both predictive performance and pedagogical quality, highlighting the positive effect of TTS on accuracy and instructional effectiveness.
  • Figure 3: Structural analysis of reasoning traces. Correct predictions are characterized by analytically centered transition structures, greater diversity, and denser transition patterns, whereas incorrect predictions exhibit more fragmented and self-referential dynamics.
  • Figure 4: Effect of history length on AUC. Thinking-KT (2048 tokens) consistently improves performance across all lengths, with saturation around 20--25 interactions.