Cold Start Problem: An Experimental Study of Knowledge Tracing Models with New Students

TL;DR

This paper investigates the cold start problem in knowledge tracing by training on historical students and evaluating on entirely new learners across $DKT$, $DKVMN$, and $SAKT$ on the ASSISTments datasets. It introduces a realistic evaluation framework and shows distinct model profiles: $DKVMN$ provides strong early accuracy and rapid gains, $DKT$ demonstrates steady long-range improvement, and $SAKT$ achieves quick initial gains but can plateau. The findings reveal persistent cold-start limitations and motivate hybrid or transfer-learning approaches to improve generalization to unseen learners, with practical implications for deploying KT in ITS. Overall, the work highlights the need for models that combine rapid adaptation with sustained learning to support diverse and evolving educational needs.

Abstract

KnowledgeTracing (KT) involves predicting students' knowledge states based on their interactions with Intelligent Tutoring Systems (ITS). A key challenge is the cold start problem, accurately predicting knowledge for new students with minimal interaction data. Unlike prior work, which typically trains KT models on initial interactions of all students and tests on their subsequent interactions, our approach trains models solely using historical data from past students, evaluating their performance exclusively on entirely new students. We investigate cold start effects across three KT models: Deep Knowledge Tracing (DKT), Dynamic Key-Value Memory Networks (DKVMN), and Self-Attentive Knowledge Tracing (SAKT), using ASSISTments 2009, 2015, and 2017 datasets. Results indicate all models initially struggle under cold start conditions but progressively improve with more interactions; SAKT shows higher initial accuracy yet still faces limitations. These findings highlight the need for KT models that effectively generalize to new learners, emphasizing the importance of developing models robust in few-shot and zero-shot learning scenarios

Figures (2)

• Figure 1: Accuracy vs Number of Questions (2009, 2015, 2017) - (a)DKT (b)DKVMN
• Figure 2: (a) SAKT Accuracy vs Number of Questions (2009, 2015, 2017) and (b) Accuracy vs Number of Questions with Different Models and Different Student Sets