LLM-KT: Aligning Large Language Models with Knowledge Tracing using a Plug-and-Play Instruction

TL;DR

This work tackles knowledge tracing by leveraging large language models to reason over rich textual knowledge and student history. It introduces LLM-KT, a plug-and-play framework that aligns LLMs with KT through a task-level Plug-and-Play Instruction and modality-level plug-ins for context and sequence, using a context encoder/adapter and a sequence encoder/adapter. Empirical results on four standard KT datasets demonstrate state-of-the-art performance, with ablations confirming the necessity of both the plug-in context and plug-in sequence components and insights into encoders and merging strategies. The approach enables effective handling of long histories and multimodal information, suggesting a practical path for deploying LLMs in personalized education with scalable alignment to KT tasks.

Abstract

The knowledge tracing (KT) problem is an extremely important topic in personalized education, which aims to predict whether students can correctly answer the next question based on their past question-answer records. Prior work on this task mainly focused on learning the sequence of behaviors based on the IDs or textual information. However, these studies usually fail to capture students' sufficient behavioral patterns without reasoning with rich world knowledge about questions. In this paper, we propose a large language models (LLMs)-based framework for KT, named \texttt{\textbf{LLM-KT}}, to integrate the strengths of LLMs and traditional sequence interaction models. For task-level alignment, we design Plug-and-Play instruction to align LLMs with KT, leveraging LLMs' rich knowledge and powerful reasoning capacity. For modality-level alignment, we design the plug-in context and sequence to integrate multiple modalities learned by traditional methods. To capture the long context of history records, we present a plug-in context to flexibly insert the compressed context embedding into LLMs using question-specific and concept-specific tokens. Furthermore, we introduce a plug-in sequence to enhance LLMs with sequence interaction behavior representation learned by traditional sequence models using a sequence adapter. Extensive experiments show that \texttt{\textbf{LLM-KT}} obtains state-of-the-art performance on four typical datasets by comparing it with approximately 20 strong baselines.

Figures (4)

• Figure 1: The advantages of traditional models and LLMs for knowledge tracing. Traditional models are good at learning the sequence of interaction behavior, while LLMs are good at reasoning with rich world knowledge.
• Figure 2: The framework of LLM-KT. We propose a Plug-and-Play Instruction to combine the strengths of LLMs and traditional sequence models for knowledge tracing by inserting multiple modalities into LLMs. Particularly, we design a Plug-in Context module to capture the long context of students' problem-solving records. Then, we introduce the Plug-in Sequence to align the sequence interaction representation learned by the traditional model with LLMs.
• Figure 3: Influence of sequence length on four different datasets in terms of AUC.
• Figure 4: Influence of sequence length on four different datasets in terms of ACC.