Unified Uncertainty Estimation for Cognitive Diagnosis Models

TL;DR

This work tackles the lack of reliable uncertainty estimates in cognitive diagnosis models (CDMs) by introducing Unified Uncertainty Estimation for Cognitive Diagnosis models (UCD). It casts CDM parameters as random variables and learns posterior distributions via a unified mini-batch objective that minimizes $D_{KL}[q(\boldsymbol\Psi|\boldsymbol\theta) || p(\boldsymbol\Psi|\mathcal{R})]$, employing a reparameterization $\boldsymbol\Psi=g(\mu+\sigma\epsilon)$ to handle diverse parameter domains. A key contribution is decomposing uncertainty into data and model components, with $\sigma_{\phi}=\sigma_{\phi}^m\cdot\sigma_{\phi}^d$ and $\sigma_{\phi}^d=\lambda_0 e^{-\lambda_1 \tau}$, enabling insight into sources of uncertainty. Through experiments on real CDMs (IRT, MIRT, NeuralCDM, KaNCD) and diverse datasets, UCD yields well-calibrated uncertainty intervals (close to 0.95 coverage) while preserving or improving diagnostic performance, and demonstrates scalability to modern deep CDMs. Overall, UCD provides a practical, uncertainty-aware framework for cognitive diagnosis with broad applicability to education and beyond.

Abstract

Cognitive diagnosis models have been widely used in different areas, especially intelligent education, to measure users' proficiency levels on knowledge concepts, based on which users can get personalized instructions. As the measurement is not always reliable due to the weak links of the models and data, the uncertainty of measurement also offers important information for decisions. However, the research on the uncertainty estimation lags behind that on advanced model structures for cognitive diagnosis. Existing approaches have limited efficiency and leave an academic blank for sophisticated models which have interaction function parameters (e.g., deep learning-based models). To address these problems, we propose a unified uncertainty estimation approach for a wide range of cognitive diagnosis models. Specifically, based on the idea of estimating the posterior distributions of cognitive diagnosis model parameters, we first provide a unified objective function for mini-batch based optimization that can be more efficiently applied to a wide range of models and large datasets. Then, we modify the reparameterization approach in order to adapt to parameters defined on different domains. Furthermore, we decompose the uncertainty of diagnostic parameters into data aspect and model aspect, which better explains the source of uncertainty. Extensive experiments demonstrate that our method is effective and can provide useful insights into the uncertainty of cognitive diagnosis.

Figures (8)

• Figure 1: A toy example.
• Figure 2: The model structures of IRT and NeuralCDM
• Figure 3: The graphic model of UCD
• Figure 4: A unidimensional illustration of interval transformation.
• Figure 5: The $\sigma_{\alpha}$ of students estimated by U-IRT and U-NCDM.

Theorems & Definitions (1)

• theorem 1