Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Does Interpretability of Knowledge Tracing Models Support Teacher Decision Making?

Adia Khalid, Alina Deriyeva, Benjamin Paassen

TL;DR

The paper examines whether interpretable KT models ($p_{t,j}$, $\theta_{t,k}$) enhance teacher decision making. Through an Elo-grounded simulation and a human-teacher study comparing BKT, PFA, and DKT, it shows that interpretable models yield faster simulated mastery and higher usability/trust, while human teachers did not consistently leverage interpretability to reduce the number of tasks to mastery. The results imply that interpretability alone does not straightforwardly improve pedagogical decisions; teachers may rely on information beyond model outputs or may misinterpret graphs, underscoring the need for better interfaces and integrative explanations. Overall, the work highlights opportunities to design KT models and interfaces that preserve interpretability while actually guiding teaching more effectively in real classrooms.

Abstract

Knowledge tracing (KT) models are a crucial basis for pedagogical decision-making, namely which task to select next for a learner and when to stop teaching a particular skill. Given the high stakes of pedagogical decisions, KT models are typically required to be interpretable, in the sense that they should implement an explicit model of human learning and provide explicit estimates of learners' abilities. However, to our knowledge, no study to date has investigated whether the interpretability of KT models actually helps human teachers to make teaching decisions. We address this gap. First, we perform a simulation study to show that, indeed, decisions based on interpretable KT models achieve mastery faster compared to decisions based on a non-interpretable model. Second, we repeat the study but ask $N=12$ human teachers to make the teaching decisions based on the information provided by KT models. As expected, teachers rate interpretable KT models higher in terms of usability and trustworthiness. However, the number of tasks needed until mastery hardly differs between KT models. This suggests that the relationship between model interpretability and teacher decisions is not straightforward: teachers do not solely rely on KT models to make decisions and further research is needed to investigate how learners and teachers actually understand and use KT models.

Does Interpretability of Knowledge Tracing Models Support Teacher Decision Making?

TL;DR

The paper examines whether interpretable KT models (, ) enhance teacher decision making. Through an Elo-grounded simulation and a human-teacher study comparing BKT, PFA, and DKT, it shows that interpretable models yield faster simulated mastery and higher usability/trust, while human teachers did not consistently leverage interpretability to reduce the number of tasks to mastery. The results imply that interpretability alone does not straightforwardly improve pedagogical decisions; teachers may rely on information beyond model outputs or may misinterpret graphs, underscoring the need for better interfaces and integrative explanations. Overall, the work highlights opportunities to design KT models and interfaces that preserve interpretability while actually guiding teaching more effectively in real classrooms.

Abstract

Knowledge tracing (KT) models are a crucial basis for pedagogical decision-making, namely which task to select next for a learner and when to stop teaching a particular skill. Given the high stakes of pedagogical decisions, KT models are typically required to be interpretable, in the sense that they should implement an explicit model of human learning and provide explicit estimates of learners' abilities. However, to our knowledge, no study to date has investigated whether the interpretability of KT models actually helps human teachers to make teaching decisions. We address this gap. First, we perform a simulation study to show that, indeed, decisions based on interpretable KT models achieve mastery faster compared to decisions based on a non-interpretable model. Second, we repeat the study but ask human teachers to make the teaching decisions based on the information provided by KT models. As expected, teachers rate interpretable KT models higher in terms of usability and trustworthiness. However, the number of tasks needed until mastery hardly differs between KT models. This suggests that the relationship between model interpretability and teacher decisions is not straightforward: teachers do not solely rely on KT models to make decisions and further research is needed to investigate how learners and teachers actually understand and use KT models.

Paper Structure

This paper contains 27 sections, 8 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. The Elo model
  4. Bayesian Knowledge Tracing (BKT)
  5. Performance Factors Analysis (PFA)
  6. Deep Knowledge Tracing (DKT)
  7. Prior studies of KT model interpretability
  8. Simulation Study
  9. Method
  10. Initialization:
  11. 1. KT prediction:
  12. 2. Task selection:
  13. 3. Sampling:
  14. 4. Simulation update:
  15. 5. KT update:
  16. ...and 12 more sections

Figures (5)

  • Figure 1: The simulation loop involving knowledge tracing, task selection, sampling of success or failure, simulation update, and KT model update.
  • Figure 2: The number of steps until mastery (left) and the number of steps until the simulation ends for the different KT models in the simulation study.
  • Figure 3: The user interface for the teacher study. Teachers could select tasks (buttons on the top left) and decide when to stop (button on the bottom right) based on which skills belonged to which tasks, past successes and failures of the simulated student, predicted success probabilities on each task by the KT model, and the ability estimates of the KT model. Best viewed in color.
  • Figure 4: The number of steps until mastery (left) and the number of steps until teaching ended (right) for the different KT models in the teacher study.
  • Figure 5: Fraction of selected tasks across time for BKT (left), PFA (center), and DKT (right) across teachers in each step of the simulation.