Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Addressing Label Leakage in Knowledge Tracing Models

Yahya Badran, Christine Preisach

TL;DR

Knowledge Tracing models that expand item interactions into KC-level sequences can suffer label leakage, where correlations among KCs of the same item reveal ground-truth labels. The authors propose a leakage-free framework with multiple leakage-prevention strategies, including mask labeling (MASK), averaged KC embeddings (DKT-Fuse), specialized attention masks (AKT-QM), and autoregressive decoding (DKT-AD), plus variants like DKT-ML and AKT-ML. Across diverse datasets with varying KC-per-item characteristics, these leakage-free methods consistently outperform their leakage-prone counterparts and competitive baselines, with AKT-ML often achieving top performance. The work underscores the practical impact of leakage in DLKT and provides KTbench as an open-source tool to support reproducibility and broader application in ITS contexts.

Abstract

Knowledge Tracing (KT) is concerned with predicting students' future performance on learning items in intelligent tutoring systems. Learning items are tagged with skill labels called knowledge concepts (KCs). Many KT models expand the sequence of item-student interactions into KC-student interactions by replacing learning items with their constituting KCs. This approach addresses the issue of sparse item-student interactions and minimises the number of model parameters. However, we identified a label leakage problem with this approach. The model's ability to learn correlations between KCs belonging to the same item can result in the leakage of ground truth labels, which leads to decreased performance, particularly on datasets with a high number of KCs per item. In this paper, we present methods to prevent label leakage in knowledge tracing (KT) models. Our model variants that utilize these methods consistently outperform their original counterparts. This further underscores the impact of label leakage on model performance. Additionally, these methods enhance the overall performance of KT models, with one model variant surpassing all tested baselines on different benchmarks. Notably, our methods are versatile and can be applied to a wide range of KT models.

Addressing Label Leakage in Knowledge Tracing Models

TL;DR

Knowledge Tracing models that expand item interactions into KC-level sequences can suffer label leakage, where correlations among KCs of the same item reveal ground-truth labels. The authors propose a leakage-free framework with multiple leakage-prevention strategies, including mask labeling (MASK), averaged KC embeddings (DKT-Fuse), specialized attention masks (AKT-QM), and autoregressive decoding (DKT-AD), plus variants like DKT-ML and AKT-ML. Across diverse datasets with varying KC-per-item characteristics, these leakage-free methods consistently outperform their leakage-prone counterparts and competitive baselines, with AKT-ML often achieving top performance. The work underscores the practical impact of leakage in DLKT and provides KTbench as an open-source tool to support reproducibility and broader application in ITS contexts.

Abstract

Knowledge Tracing (KT) is concerned with predicting students' future performance on learning items in intelligent tutoring systems. Learning items are tagged with skill labels called knowledge concepts (KCs). Many KT models expand the sequence of item-student interactions into KC-student interactions by replacing learning items with their constituting KCs. This approach addresses the issue of sparse item-student interactions and minimises the number of model parameters. However, we identified a label leakage problem with this approach. The model's ability to learn correlations between KCs belonging to the same item can result in the leakage of ground truth labels, which leads to decreased performance, particularly on datasets with a high number of KCs per item. In this paper, we present methods to prevent label leakage in knowledge tracing (KT) models. Our model variants that utilize these methods consistently outperform their original counterparts. This further underscores the impact of label leakage on model performance. Additionally, these methods enhance the overall performance of KT models, with one model variant surpassing all tested baselines on different benchmarks. Notably, our methods are versatile and can be applied to a wide range of KT models.
Paper Structure (19 sections, 16 equations, 6 figures, 2 tables)

This paper contains 19 sections, 16 equations, 6 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. KNOWLEDGE TRACING
  4. Review of Deep Knowledge Tracing
  5. Review of Attentive Knowledge Tracing
  6. PROBLEM STATEMENT
  7. Evaluation Problems
  8. Training Problems
  9. LABEL LEAKAGE-FREE FRAMEWORK
  10. Incorporating a Mask Label
  11. AKT-ML
  12. DKT-ML
  13. DKT with Averaged Embeddings
  14. Special Attention Masks
  15. Autoregressive Decoding of the DKT Model
  16. ...and 4 more sections

Figures (6)

  • Figure 1: Expanding a question-student interaction sequence into a KC-student interaction sequence. The green and red symbols are correct and incorrect respectively
  • Figure 2: Overview of the AKT model architecture. This is a simplified version, some blocks are repeated. Each attention block in the figure takes the sequence of inputs---value, query, and key---from left to right.
  • Figure 3: one-by-one evaluation and training on the expanded sequences. Note, $c_5$ ground-truth label can leak to $r_6'$ as both $c_5$ and $c_6$ belong to the same question, $q_3$.
  • Figure 4: The all-in-one evaluation method. Both predictions of $c_5$ and $c_6$ should be produced independently of each other as they belong to the same question, $q_3$.
  • Figure 5: Expanding question-student interaction sequence into KC-student interaction sequence with MASK labels. The green and red symbols are correct and incorrect respectively
  • ...and 1 more figures