Knowledge is Power: Harnessing Large Language Models for Enhanced Cognitive Diagnosis

TL;DR

The paper targets the cold-start problem in cognitive diagnosis by marrying large language models (LLMs) with cognitive diagnosis models (CDMs) through Knowledge-enhanced Cognitive Diagnosis (KCD). It introduces a two-stage framework: LLM Diagnosis, which uses prompts to generate textual diagnoses of students and exercises from collaborative information and response logs, and Cognitive Level Alignment, which bridges the semantic space of LLMs with the behavioral space of CDMs via behavioral-space contrastive learning (KCD-Beh) and semantic-space mask-reconstruction (KCD-Sem). Through global and local contrastive losses and a dynamic masking strategy, the method improves diagnostic accuracy across multiple CDMs and datasets, with pronounced gains in cold-start scenarios. The results, supported by ablations and visualizations, demonstrate effective semantic-behavioral alignment and practical improvements, underscoring the potential of LLM-informed CDMs for robust, scalable educational diagnostics; code and data are made available for reproducibility.

Abstract

Cognitive Diagnosis Models (CDMs) are designed to assess students' cognitive states by analyzing their performance across a series of exercises. However, existing CDMs often struggle with diagnosing infrequent students and exercises due to a lack of rich prior knowledge. With the advancement in large language models (LLMs), which possess extensive domain knowledge, their integration into cognitive diagnosis presents a promising opportunity. Despite this potential, integrating LLMs with CDMs poses significant challenges. LLMs are not well-suited for capturing the fine-grained collaborative interactions between students and exercises, and the disparity between the semantic space of LLMs and the behavioral space of CDMs hinders effective integration. To address these issues, we propose a novel Knowledge-enhanced Cognitive Diagnosis (KCD) framework, which is a model-agnostic framework utilizing LLMs to enhance CDMs and compatible with various CDM architectures. The KCD framework operates in two stages: LLM Diagnosis and Cognitive Level Alignment. In the LLM Diagnosis stage, both students and exercises are diagnosed to achieve comprehensive and detailed modeling. In the Cognitive Level Alignment stage, we bridge the gap between the CDMs' behavioral space and the LLMs' semantic space using contrastive learning and mask-reconstruction approaches. Experiments on several real-world datasets demonstrate the effectiveness of our proposed framework.

Figures (6)

• Figure 1: (a) An illustration of cognitive diagnosis. (b) Performance of warm and cold scenarios of NCD on PTADisc, exhibiting the limitations in cold scenario.
• Figure 2: Framework overview. (a) LLM Diagnosis generates diagnoses for students and exercises using LLMs. (b) Cognitive Level Alignment integrates LLMs and CDMs to model students and exercises in both semantic space and behavioral space.
• Figure 3: Performance comparison in cold (blue) and warm (red) scenarios on Python dataset.
• Figure 4: Performance on different dropout ratios.
• Figure 5: The t-SNE visualization of student embeddings on Literature dataset.