ReliCD: A Reliable Cognitive Diagnosis Framework with Confidence Awareness

TL;DR

ReliCD tackles the overconfidence gap in cognitive diagnostics caused by noisy, sparse interaction data by introducing a Bayesian, confidence-aware framework that models per-student state uncertainty with Gaussian latent representations. It supports multiple diagnostic functions (IRT, MIRT, NCD) and adds prior-consensus initialization plus a calibration loss to rank and regularize diagnostic confidence. The approach achieves lower calibration errors (ECE/MCE) and competitive predictive metrics across four real-world datasets, with Ablation and sensitivity analyses highlighting the value of pre-training and the calibration loss. The work advances practical educational AI by providing reliable, explainable diagnostic feedback that can guide interventions and interpretation in real-time settings.

Abstract

During the past few decades, cognitive diagnostics modeling has attracted increasing attention in computational education communities, which is capable of quantifying the learning status and knowledge mastery levels of students. Indeed, the recent advances in neural networks have greatly enhanced the performance of traditional cognitive diagnosis models through learning the deep representations of students and exercises. Nevertheless, existing approaches often suffer from the issue of overconfidence in predicting students' mastery levels, which is primarily caused by the unavoidable noise and sparsity in realistic student-exercise interaction data, severely hindering the educational application of diagnostic feedback. To address this, in this paper, we propose a novel Reliable Cognitive Diagnosis(ReliCD) framework, which can quantify the confidence of the diagnosis feedback and is flexible for different cognitive diagnostic functions. Specifically, we first propose a Bayesian method to explicitly estimate the state uncertainty of different knowledge concepts for students, which enables the confidence quantification of diagnostic feedback. In particular, to account for potential differences, we suggest modeling individual prior distributions for the latent variables of different ability concepts using a pre-trained model. Additionally, we introduce a logical hypothesis for ranking confidence levels. Along this line, we design a novel calibration loss to optimize the confidence parameters by modeling the process of student performance prediction. Finally, extensive experiments on four real-world datasets clearly demonstrate the effectiveness of our ReliCD framework.

Figures (9)

• Figure 1: (a) An example of cognitive diagnosis; (b) the predicted Lano's diagnostic feedback on concept $C_2$ with different interaction data and the corresponding accuracy of her performance prediction on all the exercises related to the concept $C_2$ in the test set, where $e_{1:j}$ denotes the exercises set $\{e_1, e_2, ..., e_j\}$ and $\bar{h}$ indicates Lano's actual ability state on $C_2$.
• Figure 2: (a) The distribution of all students' ability status diagnosed by NCD on the Assist2009 dataset. The blue part represents diagnostic status of knowledge concepts not interacted with, and the red part represents diagnostic status of knowledge concepts interacted with. (b) The density plot of all students' status on the knowledge concepts that they have interacted with.
• Figure 3: (a) The density plot of the correct rate of students' performance prediction task related to knowledge concept #50 by NCD in the test set of Assist2009. (b) The density plot of the correct rate after randomly adding one noisy interaction data on concept #50 for each student.
• Figure 4: The illustration of our basic idea in ReliCD. Each student $s_i$ is denoted by a personalized Gaussian distribution variable $z_i\sim \mathcal{N}(\mu_i,\sigma^2_i)$ and the corresponding ability state $\theta_i$ can be specified by applying the Sigmoid function on $z_i$, which is also a distribution with the support on $[0,1]$. Next, prior common cognition $\mathcal{N}(\mu_{mean}, 1)$ helps avoid the situation that students master all knowledge concepts in advance to 0. Then, a calibration loss is induced to close the relationship between uncertainty and the reliability of the student's ability states by establishing a ranking relationship.
• Figure 5: Results of Reli-NCD and its variants.

Theorems & Definitions (1)

• Definition 1