Conformal Depression Prediction

TL;DR

The paper tackles the challenge of trustworthy uncertainty estimation in deep depression prediction by introducing Conformal Depression Prediction (CDP), a distribution-free method that delivers valid confidence intervals with marginal coverage without requiring retraining. It extends CDP with CDP-ACC, an adaptive approach that achieves approximate conditional coverage by leveraging neighborhood-based histograms in the predicted-score space to produce input-specific intervals. Through experiments on AVEC 2013 and AVEC 2014 facial depression datasets, CDP demonstrates reliable, valid intervals, while CDP-ACC yields tighter, more informative intervals with competitive conditional coverage. The work provides a practical, extensible framework for uncertainty quantification in depression prediction, with public code and potential applicability to other data modalities.

Abstract

While existing depression prediction methods based on deep learning show promise, their practical application is hindered by the lack of trustworthiness, as these deep models are often deployed as black box models, leaving us uncertain on the confidence of their predictions. For high-risk clinical applications like depression prediction, uncertainty quantification is essential in decision-making. In this paper, we introduce conformal depression prediction (CDP), a depression prediction method with uncertainty quantification based on conformal prediction (CP), giving valid confidence intervals with theoretical coverage guarantees for the model predictions. CDP is a plug-and-play module that requires neither model retraining nor an assumption about the depression data distribution. As CDP provides only an average coverage guarantee across all inputs rather than per-input performance guarantee, we further propose CDP-ACC, an improved conformal prediction with approximate conditional coverage. CDP-ACC firstly estimates the prediction distribution through neighborhood relaxation, and then introduces a conformal score function by constructing nested sequences, so as to provide a tighter prediction interval adaptive to specific input. We empirically demonstrate the application of CDP in uncertainty-aware facial depression prediction, as well as the effectiveness and superiority of CDP-ACC on the AVEC 2013 and AVEC 2014 datasets. Our code is publicly available at https://github.com/PushineLee/CDP.

Figures (8)

• Figure 1: Uncertainty quantification for facial depression prediction can provide an additional confidence interval to score prediction at any given confidence level, such as 90%.
• Figure 2: The pipeline of CDP. CDP takes as input a pre-trained depression prediction model $\hat{f}$, a calibration set $\mathcal{D}_{cal}$ and a given confidence level $1 - \alpha$. CDP sets the prediction biases (on the calibration set) as the conformal scores and employs the $(1 - \alpha)$-th empirical quantile of the conformal scores to form confidence interval for the prediction $\hat{f}(X_{test})$.
• Figure 3: The pipeline of CDP-ACC. The predictions on the calibration set are firstly partitioned into $M$ subintervals. For each subinterval, CDP-ACC estimates the approximate conditional distribution $P(y|\hat{y})$, computes the conformal score based on the calibration samples in that interval, and finally constructs the adaptive confidence interval.
• Figure 4: Histogram estimation of the conditional distribution $P(y| \hat{y}\in \Delta(\hat{y}))$. For samples falling in $\hat{y} \in \Delta(\hat{y})$, the more concentrated the points, the higher the credibility.
• Figure 5: The setting of the conformal score for CDP-ACC. Given the approximate conditional distribution $P(y|\hat{y})$, various prediction intervals can be acquired containing $y_i$ with $1 - \alpha$ coverage, of which the width of the shortest one is set as the conformal score for our CDP-ACC.