Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncertainty-Aware Knowledge Tracing Models

Joshua Mitton, Prarthana Bhattacharyya, Ralph Abboud, Simon Woodhead

TL;DR

This paper tackles the lack of uncertainty quantification in Knowledge Tracing (KT) models, which hampers detection of student misconceptions. It extends KT to a multi-class, distractor-aware setting and applies Monte Carlo Dropout to four KT architectures to quantify predictive uncertainty. Empirical results show that predictive uncertainty correlates with mispredictions and that entropy and standard deviation signals provide useful cues for pedagogy, even on large math datasets. The work demonstrates that uncertainty-aware, attention-based models with text-rich embeddings yield calibrated, actionable signals for instructional decisions in resource-constrained educational platforms.

Abstract

The main focus of research on Knowledge Tracing (KT) models is on model developments with the aim of improving predictive accuracy. Most of these models make the most incorrect predictions when students choose a distractor, leading to student errors going undetected. We present an approach to add new capabilities to KT models by capturing predictive uncertainty and demonstrate that a larger predictive uncertainty aligns with model incorrect predictions. We show that uncertainty in KT models is informative and that this signal would be pedagogically useful for application in an educational learning platform that can be used in a limited resource setting where understanding student ability is necessary.

Uncertainty-Aware Knowledge Tracing Models

TL;DR

This paper tackles the lack of uncertainty quantification in Knowledge Tracing (KT) models, which hampers detection of student misconceptions. It extends KT to a multi-class, distractor-aware setting and applies Monte Carlo Dropout to four KT architectures to quantify predictive uncertainty. Empirical results show that predictive uncertainty correlates with mispredictions and that entropy and standard deviation signals provide useful cues for pedagogy, even on large math datasets. The work demonstrates that uncertainty-aware, attention-based models with text-rich embeddings yield calibrated, actionable signals for instructional decisions in resource-constrained educational platforms.

Abstract

The main focus of research on Knowledge Tracing (KT) models is on model developments with the aim of improving predictive accuracy. Most of these models make the most incorrect predictions when students choose a distractor, leading to student errors going undetected. We present an approach to add new capabilities to KT models by capturing predictive uncertainty and demonstrate that a larger predictive uncertainty aligns with model incorrect predictions. We show that uncertainty in KT models is informative and that this signal would be pedagogically useful for application in an educational learning platform that can be used in a limited resource setting where understanding student ability is necessary.

Paper Structure

This paper contains 18 sections, 3 equations, 8 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Predictive Uncertainty for Knowledge Tracing Models
  4. Knowledge Tracing Models
  5. Baseline KT Models
  6. Limitations of KT Models
  7. MCDropout for KT Uncertainty
  8. Experiments and Results
  9. Uncertainty Prediction in Knowledge Tracing Models
  10. How Do Different Models Capture Uncertainty?
  11. Conclusion
  12. Disclosure of Interests.
  13. Experimental Setup
  14. Dataset Details
  15. Model Training
  16. ...and 3 more sections

Figures (8)

  • Figure 1: Total entropy distribution of each model's predictions as a function of their correctness. Misclassified data points clearly demonstrate a higher level of uncertainty.
  • Figure 2: Box plot of total entropy of each model's predictions split into whether the student chose a correct response or an incorrect response.
  • Figure 3: Box plot of each model's prediction standard deviation split into whether the student chose a correct response or an incorrect response.
  • Figure 4: Comparison of each models predictive total entropy against the question number currently being predicted.
  • Figure 5: Comparison of each models prediction standard deviation against the question number currently being predicted.
  • ...and 3 more figures