LLM4CD: Leveraging Large Language Models for Open-World Knowledge Augmented Cognitive Diagnosis

TL;DR

Cognition diagnosis in ITS has been hampered by reliance on ID embeddings that ignore semantic context and by cold-start when new students or exercises appear. LLM4CD introduces a bi-level encoder that leverages open-world knowledge from large language models to construct semantically rich representations, complemented by macro-level text encoding and micro-level state encoding to adapt to unseen data. The framework builds comprehensive relation graphs over concepts and exercises, enabling semantically informed inference and robust predictions across multiple real-world datasets, with ablations highlighting the critical role of semantic embeddings and macro-text information. The approach yields consistent performance gains, demonstrates improved cold-start handling, and provides interpretable diagnostic reports, suggesting practical improvements for open-world, text-rich cognitive diagnosis in modern ITS.

Abstract

Cognitive diagnosis (CD) plays a crucial role in intelligent education, evaluating students' comprehension of knowledge concepts based on their test histories. However, current CD methods often model students, exercises, and knowledge concepts solely on their ID relationships, neglecting the abundant semantic relationships present within educational data space. Furthermore, contemporary intelligent tutoring systems (ITS) frequently involve the addition of new students and exercises, a situation that ID-based methods find challenging to manage effectively. The advent of large language models (LLMs) offers the potential for overcoming this challenge with open-world knowledge. In this paper, we propose LLM4CD, which Leverages Large Language Models for Open-World Knowledge Augmented Cognitive Diagnosis. Our method utilizes the open-world knowledge of LLMs to construct cognitively expressive textual representations, which are then encoded to introduce rich semantic information into the CD task. Additionally, we propose an innovative bi-level encoder framework that models students' test histories through two levels of encoders: a macro-level cognitive text encoder and a micro-level knowledge state encoder. This approach substitutes traditional ID embeddings with semantic representations, enabling the model to accommodate new students and exercises with open-world knowledge and address the cold-start problem. Extensive experimental results demonstrate that our proposed method consistently outperforms previous CD models on multiple real-world datasets, validating the effectiveness of leveraging LLMs to introduce rich semantic information into the CD task.

Figures (7)

• Figure 1: (a) An illustration of the cognitive diagnosis (CD) task with semantic concepts; (b) An example of the cold-start problem on traditional ID-based CD models.
• Figure 2: The overall framework of LLM4CD.
• Figure 3: Text construction from the exercise and student sides.
• Figure 4: Performance on cold-start prediction for new students in CD models on four datasets. The Length of Sequence $K$ represents the number of interactions per student that have not been seen during training.
• Figure 5: Comparison of different text representations from the student side.